Jump to content

How to migrate your data from 1 KVS site to another

Tech Support

Recommended Posts

Migrating categorization

In most cases you don't need to consider categorization, as it will be automatically migrated along with content. However if you want to migrate categorization details as well (such as model details, screenshots, or custom fields) the only way to do that is to use Migrator plugin in KVS.

Migrator plugin will connect to another KVS database and will pull categorization that you choose to pull. There are certain important aspects that you should consider when using Migrator plugin:

  • Migrator plugin only supports categorization for KVS migrator. Content migration should be done using export and import functionality, which has much more options.
  • Migrator plugin needs direct connection to other project's database. This works fine if your other project is on the same server, but won't work if your projects are on different servers. One option is to ask admins to expose KVS database on the other project's server to be available for public connections, which is not good from security point. Another option is to create a temporary local database and import backup from the other project's database into it, then configure Migrator plugin to migrate from this local temporary database.
  • Migrator plugin needs your new database to be fully empty (e.g. no new categorization objects in it), as it will migrate objects by using the same IDs and will replace any existing objects that have the same IDs that are migrated from the other project's database. WARNING! If you try using Migrator plugin on project where you already have some categorization, it can be lost.
  • It is recommended that both projects have the same KVS version before you try migration. Otherwise it is not guaranteed to fully succeed.

Having these in mind, if your other project is on the same server and your new project is just installed, you can easily pull up all categories, tags, models, content sources with all their data within 5 minutes. All you need is to specify database connection to the other project's database. If you don't know MySQL connection details, please open /admin/include/setup_db.php file to find them. Here is an example:


NOTE: for the Old path please specify any existing empty directory path, you can specify path to /tmp directory in your www root.


Manually migrating video or album content

Use manual content export / import when you want to do a one time migration of all (or part) of your content from older project and then allow these projects run separately. If you want to constantly sync one project based on another project, you need to use feeds, which are explained in the next section.

The idea is that you first export content data into a text file and then use this text file to import the exported content into another project. Export / import of albums and videos works in exactly same way, so we will describe on videos.

You can access export function in Videos -> Export videos (Import videos for import). The important thing is to create a named pattern of export (and later import), so that you can return to it later if you do any mistakes with fields or options. The recommended way is to first export a dozen of test videos  and then try importing them to test the whole scheme and that you get the result you need; and only after you are fine with the imported content you can do the full migration.

Another important thing to consider is which video content types you have (file uploads, embeds, hotlinks, or mixed) and how you want them to be imported. The most common scenario is that you have file uploads (videos that store files in KVS storage) and you want to migrate them in the same way. Another possible scenario is that you have videos embedded from other sites and you want to migrate them in the same way. In these cases when you have ALL videos of the same content type, you just do one time export of them ALL and one time import of them ALL. However, there are rare cases when you have some videos uploaded to your servers, some more videos embedded and another set of videos just hotlinked. In this specific case you should do separate export and import for each content type that you have. This is needed because videos of different content types have different data structure. Video export GUI offers Loaded as filter that allows you separating videos by their content type.

With regard to Field and Line separators it is recommended to leave them as default. This may not work only if you have videos with multiline descriptions, but in this case manual export and import won't work for you at all and you have to fallback to using KVS feeds.

In general it is recommended to leave all export and import options to their default values, unless you need anything specific. The main area where you should put your effort is Fields for export (Fields for import). First of all you should be using the same Field and Line separators in both export and import, we recommend use them as set by default. Next you should make sure that you select the same sets of fields in both export and import. Many fields like title and description are obvious, how about the rest of them? Here is a short summary:

  • Use Video ID only if you want to keep the same IDs across 2 projects. In most cases this is not needed.
  • Use Directory only if you had some specific rules on directory generation or transliteration. In most case this is not needed as directory is usually generated from title.
  • If you want to migrate rating, make sure you select both Rating and Rating (votes).
  • If you want to migrate usernames of video creators, you can do that by selecting User field, but please note that this will only migrate the username, not the whole user data. KVS will create new fake users with the given usernames in new projects.

Finally there is a set of fields under Content group, which are related to content structure. As we noted before, depending on video content type these fields structure will be different:

  • For file upload content type, you should first get understanding which video formats you want to migrate. In many cases video formats on your new site will be configured in the same way as on your old site, so you will need to migrate all video formats. But in some cases your new site may have less formats, for example it could have no video preview format, or you occasionally want video preview to be generated differently on this new site. Based on this understanding you should export Video file "XXX" (for hotlink) field for each of the video formats you want to migrate. At the same time you should select the corresponding Video file "XXX" (without processing) field in import fields configuration.
  • For embed content type, you should export and import these fields: Embed code (embedded video), Duration, Main screenshot number and Overview screenshots (sources comma-separated).
  • For hotlink content type, you should export and import these fields: Hosted file URL (hotlinking), Duration, Main screenshot number and Overview screenshots (sources comma-separated).
  • For pseudo video content type, you should export and import these fields: Outgoing URL (pseudo video), Duration, Main screenshot number and Overview screenshots (sources comma-separated).

NOTE: if you have mixed content types, remember to do separate export and import for each content type with different set of content fields.

NOTE: change Sorting to Video ID Asc so that videos are exported and imported in the same order as on the original site.

IMPORTANT! When importing uploaded videos you should make sure that source site has video download protection disabled for the whole period of import. You can disable this in Settings -> Content settings -> Enable protection for video download script option. After import is fully completed, please enable it back.

With regard to screenshots, for file uploads they are optional and it is up to you if you want to migrate them or not. In any way KVS will create new screenshots if needed. When migrating screenshots, please make sure that you export their source files, not files of any specific format. And also make sure you migrate main screenshot number.

Here is example of how we selected fields in export and import to migrate some uploaded videos from one site to another:



Automating content sync with feed

You only need to use feeds if you plan to repeatedly sync other sites from the content of you primary site, or in case descriptions of your videos may contain line breaks, so that it is not possible to normally transfer them via manual CSV import. For a one time content transfer it is better to use manual import, as it provides more options and features.

It also worth mentioning that only KVS Ultimate supports exporting feeds, so your primary project should have ultimate license. Importing feeds are supported in all KVS licenses. Also at the moment KVS doesn't support feeds for albums.

Part 1: Exporting feed.

You should start with creating exporting feed in your primary project, you can do that in Videos -> Exporting feeds. The most important section is Feed data and its Content type option. This option indicates which content details you want to expose in feed, and this depends on how you plan to sync your content into another KVS project (please find several examples of how to create site network from a single KVS site in How to build your tube empire with KVS article):

  • Use Website link option if you want to migrate these videos as pseudo videos.
  • Use Hotlink option if you plan to hotlink video files from this project on another projects.
  • Use Embed option if you want to embed these videos on your another projects.
  • Finally use Download link option if you are going to fully sync video files of videos of this project to your another projects. IMPORTANT! When using this option you should make sure that source site has video download protection disabled for the whole period of import. You can disable this in Settings -> Content settings -> Enable protection for video download script option. After import is fully completed, please enable it back.

For other options make sure to enable all categorization data, screenshot and poster sources, custom and translated fields. Increase Max limit option to 10000, or bigger value that you want to use (see explanation below).

If you plan to use embed codes or pseudo videos and you need some tracking of all traffic that comes to your primary site, you may also need to set some Traffic tracking parameters that you can later specify in feed builder.

After your new exporting feed is saved, open its Feed access point to see feed builder form that provides ability to query the needed data from the feed. You should have Feed format set to KVS. Use Filters section if you want to sync only subset of content. If you are going to embed videos on another projects, check URLs and embeds section options. You will also need to specify details for any Traffic tracking parameters there if you enabled them. If you plan to hotlink video files on another projects, check Video files section that provides ability to select either quality factor for hotlinked URLs, or choose a specific video format to hotlink. In most cases you won't need anything specific for download mode, as it works fine by default.

Another important thing to consider at this point is how many videos your feed would contain. If your database has less than 10k videos, you won't face any issue in most cases. More videos mean more memory and database load to generate all feed data, which will reach red line at one point and feed will stop working or hang for a long time. In order to solve this, you should use feed limit and pagination control to paginate over the whole list. E.g. import the first 10k videos with the first iteration, then import another 10k videos, and etc. In order to do that you should generate all feed URLs once, don't leave it for the latter time, as later you may forget which options you selected.

Change Sorting to Biggest ID last so that videos are exported in the order as they were added.

First specify Limit = 10000 and press Get videos button. It should open another tab with feed URL that queries the first 10k videos. Copy URL and save it to a text document. Then in feed constructor set Skip to 10000 and press Get videos button again. You will get another URL that queries the second 10k videos. Repeat the same by incrementing Skip by 10000 again and again until you finally get empty list that will indicate end of paginated data. You can try increasing limit to 20000 and see if feed works with more videos, so that you do less iterations.

This will give you some set of URLs to query all your existing videos. Finally you need the last URL that will query some set of new videos for repeated sync. Change Sorting to Most recent first to make sure that now only new videos are in the feed. Set Limit to 1000 now and get URL for querying most recent 1000 videos that you will later use to sync new videos.

Part 2: Importing feeds.

Now you should go to admin panels of sites that you want to pull videos from your exporting feeds. Go to Videos -> Importing feeds and create a new importing feed. In Feed URL set the first URL of your URLs list that you created when using feed builder. Choose KVS (XML) parser. In Duplicates prefix specify domain name of your primary project you export videos from. If you create multiple importing feeds for the same project, please make sure that all share that same duplicates prefix, so that's why the best idea is to use source project domain name here.

Under Interval field choose Run this feed only once and deactivate. Since you have paginated data (if source site contains more than 10k videos you will most likely end up with feed data split into 10k-volume pages), you will need to import each page separately. The proposed way is to first configure feed to import all videos from the first page, then wait for it to be deactivated and re-configure to import from the second page URL, and etc. until you import all pages with all existing videos. After that you should change Feed URL to the last URL you created that will pull latest 1000 videos and configure its Interval to how often you want feed to pull new videos. Then feed will automatically pull new videos for the rest of time.

Under Feed data check all data fields you want to import. In Video data section set Video status to Active, Video screenshots to Use screenshots from feed, Video publishing dates to Use publishing date from feed.

Finally in Video adding mode option you should choose whether you want to embed, or hotlink, or download videos. If you want to download, then in most cases you will need to set All formats that have match by postfixes (without processing) in Format option. However in order this option to work correctly, you should go to Settings -> Video formats in this project and make sure the formats are configured in the same way they are configured on source site. At least the formats that you want to be downloaded via feeds. When we say "in the same way", we mean that their postfixes and statuses should be set accordingly. For example if on source project you have 480p, 720p formats and MP4 preview format, then you should make sure your importing site also have these formats with the same postfixes. If you don't want, for example, to download MP4 preview format and you want it to be created on the importing project with some other options - create it with another postfix syntax and required status. Then importing feed should download 480p and 720p files only and KVS will auto-create your custom MP4 preview from them.

NOTE: for testing purposes it is recommended to set Max videos per execution to some small number (e.g. 10-100) for the first iteration to make sure you configured importing feed correctly. Then you can remove this limit and run feed again allowing it importing the whole set of data.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...