Search the Community
Showing results for tags 'feeds'.
Hello, I am new to KVS and playing around with it's features. I ended up with my search but couldn't find this particular option not sure if it is already available or not, so creating this thread. Sometimes we just need to change the URL of the source feed provider and rest every field are usually same for example if I am adding 10 feeds from a source with 10 different cats, I only need to change the feed url. Can you please tell me if it is possible to copy already imported feed and just change its name/title and the URL without adding and matching every required field. Or if there any other easy way to do this? Thanks
Migrating categorization In most cases you don't need to consider categorization, as it will be automatically migrated along with content. However if you want to migrate categorization details as well (such as model details, screenshots, or custom fields) the only way to do that is to use Migrator plugin in KVS. Migrator plugin will connect to another KVS database and will pull categorization that you choose to pull. There are certain important aspects that you should consider when using Migrator plugin: Migrator plugin only supports categorization for KVS migrator. Content migration should be done using export and import functionality, which has much more options. Migrator plugin needs direct connection to other project's database. This works fine if your other project is on the same server, but won't work if your projects are on different servers. One option is to ask admins to expose KVS database on the other project's server to be available for public connections, which is not good from security point. Another option is to create a temporary local database and import backup from the other project's database into it, then configure Migrator plugin to migrate from this local temporary database. Migrator plugin needs your new database to be fully empty (e.g. no new categorization objects in it), as it will migrate objects by using the same IDs and will replace any existing objects that have the same IDs that are migrated from the other project's database. WARNING! If you try using Migrator plugin on project where you already have some categorization, it can be lost. It is recommended that both projects have the same KVS version before you try migration. Otherwise it is not guaranteed to fully succeed. Having these in mind, if your other project is on the same server and your new project is just installed, you can easily pull up all categories, tags, models, content sources with all their data within 5 minutes. All you need is to specify database connection to the other project's database. If you don't know MySQL connection details, please open /admin/include/setup_db.php file to find them. Here is an example: NOTE: for the Old path please specify any existing empty directory path, you can specify path to /tmp directory in your www root. Manually migrating video or album content Use manual content export / import when you want to do a one time migration of all (or part) of your content from older project and then allow these projects run separately. If you want to constantly sync one project based on another project, you need to use feeds, which are explained in the next section. The idea is that you first export content data into a text file and then use this text file to import the exported content into another project. Export / import of albums and videos works in exactly same way, so we will describe on videos. You can access export function in Videos -> Export videos (Import videos for import). The important thing is to create a named pattern of export (and later import), so that you can return to it later if you do any mistakes with fields or options. The recommended way is to first export a dozen of test videos and then try importing them to test the whole scheme and that you get the result you need; and only after you are fine with the imported content you can do the full migration. Another important thing to consider is which video content types you have (file uploads, embeds, hotlinks, or mixed) and how you want them to be imported. The most common scenario is that you have file uploads (videos that store files in KVS storage) and you want to migrate them in the same way. Another possible scenario is that you have videos embedded from other sites and you want to migrate them in the same way. In these cases when you have ALL videos of the same content type, you just do one time export of them ALL and one time import of them ALL. However, there are rare cases when you have some videos uploaded to your servers, some more videos embedded and another set of videos just hotlinked. In this specific case you should do separate export and import for each content type that you have. This is needed because videos of different content types have different data structure. Video export GUI offers Loaded as filter that allows you separating videos by their content type. With regard to Field and Line separators it is recommended to leave them as default. This may not work only if you have videos with multiline descriptions, but in this case manual export and import won't work for you at all and you have to fallback to using KVS feeds. In general it is recommended to leave all export and import options to their default values, unless you need anything specific. The main area where you should put your effort is Fields for export (Fields for import). First of all you should be using the same Field and Line separators in both export and import, we recommend use them as set by default. Next you should make sure that you select the same sets of fields in both export and import. Many fields like title and description are obvious, how about the rest of them? Here is a short summary: Use Video ID only if you want to keep the same IDs across 2 projects. In most cases this is not needed. Use Directory only if you had some specific rules on directory generation or transliteration. In most case this is not needed as directory is usually generated from title. If you want to migrate rating, make sure you select both Rating and Rating (votes). If you want to migrate usernames of video creators, you can do that by selecting User field, but please note that this will only migrate the username, not the whole user data. KVS will create new fake users with the given usernames in new projects. Finally there is a set of fields under Content group, which are related to content structure. As we noted before, depending on video content type these fields structure will be different: For file upload content type, you should first get understanding which video formats you want to migrate. In many cases video formats on your new site will be configured in the same way as on your old site, so you will need to migrate all video formats. But in some cases your new site may have less formats, for example it could have no video preview format, or you occasionally want video preview to be generated differently on this new site. Based on this understanding you should export Video file "XXX" (for hotlink) field for each of the video formats you want to migrate. At the same time you should select the corresponding Video file "XXX" (without processing) field in import fields configuration. For embed content type, you should export and import these fields: Embed code (embedded video), Duration, Main screenshot number and Overview screenshots (sources comma-separated). For hotlink content type, you should export and import these fields: Hosted file URL (hotlinking), Duration, Main screenshot number and Overview screenshots (sources comma-separated). For pseudo video content type, you should export and import these fields: Outgoing URL (pseudo video), Duration, Main screenshot number and Overview screenshots (sources comma-separated). NOTE: if you have mixed content types, remember to do separate export and import for each content type with different set of content fields. NOTE: change Sorting to Video ID Asc so that videos are exported and imported in the same order as on the original site. IMPORTANT! When importing uploaded videos you should make sure that source site has video download protection disabled for the whole period of import. You can disable this in Settings -> Content settings -> Enable protection for video download script option. After import is fully completed, please enable it back. With regard to screenshots, for file uploads they are optional and it is up to you if you want to migrate them or not. In any way KVS will create new screenshots if needed. When migrating screenshots, please make sure that you export their source files, not files of any specific format. And also make sure you migrate main screenshot number. Here is example of how we selected fields in export and import to migrate some uploaded videos from one site to another: Automating content sync with feed You only need to use feeds if you plan to repeatedly sync other sites from the content of you primary site, or in case descriptions of your videos may contain line breaks, so that it is not possible to normally transfer them via manual CSV import. For a one time content transfer it is better to use manual import, as it provides more options and features. It also worth mentioning that only KVS Ultimate supports exporting feeds, so your primary project should have ultimate license. Importing feeds are supported in all KVS licenses. Also at the moment KVS doesn't support feeds for albums. Part 1: Exporting feed. You should start with creating exporting feed in your primary project, you can do that in Videos -> Exporting feeds. The most important section is Feed data and its Content type option. This option indicates which content details you want to expose in feed, and this depends on how you plan to sync your content into another KVS project (please find several examples of how to create site network from a single KVS site in How to build your tube empire with KVS article): Use Website link option if you want to migrate these videos as pseudo videos. Use Hotlink option if you plan to hotlink video files from this project on another projects. Use Embed option if you want to embed these videos on your another projects. Finally use Download link option if you are going to fully sync video files of videos of this project to your another projects. IMPORTANT! When using this option you should make sure that source site has video download protection disabled for the whole period of import. You can disable this in Settings -> Content settings -> Enable protection for video download script option. After import is fully completed, please enable it back. For other options make sure to enable all categorization data, screenshot and poster sources, custom and translated fields. Increase Max limit option to 10000, or bigger value that you want to use (see explanation below). If you plan to use embed codes or pseudo videos and you need some tracking of all traffic that comes to your primary site, you may also need to set some Traffic tracking parameters that you can later specify in feed builder. After your new exporting feed is saved, open its Feed access point to see feed builder form that provides ability to query the needed data from the feed. You should have Feed format set to KVS. Use Filters section if you want to sync only subset of content. If you are going to embed videos on another projects, check URLs and embeds section options. You will also need to specify details for any Traffic tracking parameters there if you enabled them. If you plan to hotlink video files on another projects, check Video files section that provides ability to select either quality factor for hotlinked URLs, or choose a specific video format to hotlink. In most cases you won't need anything specific for download mode, as it works fine by default. Another important thing to consider at this point is how many videos your feed would contain. If your database has less than 10k videos, you won't face any issue in most cases. More videos mean more memory and database load to generate all feed data, which will reach red line at one point and feed will stop working or hang for a long time. In order to solve this, you should use feed limit and pagination control to paginate over the whole list. E.g. import the first 10k videos with the first iteration, then import another 10k videos, and etc. In order to do that you should generate all feed URLs once, don't leave it for the latter time, as later you may forget which options you selected. Change Sorting to Biggest ID last so that videos are exported in the order as they were added. First specify Limit = 10000 and press Get videos button. It should open another tab with feed URL that queries the first 10k videos. Copy URL and save it to a text document. Then in feed constructor set Skip to 10000 and press Get videos button again. You will get another URL that queries the second 10k videos. Repeat the same by incrementing Skip by 10000 again and again until you finally get empty list that will indicate end of paginated data. You can try increasing limit to 20000 and see if feed works with more videos, so that you do less iterations. This will give you some set of URLs to query all your existing videos. Finally you need the last URL that will query some set of new videos for repeated sync. Change Sorting to Most recent first to make sure that now only new videos are in the feed. Set Limit to 1000 now and get URL for querying most recent 1000 videos that you will later use to sync new videos. Part 2: Importing feeds. Now you should go to admin panels of sites that you want to pull videos from your exporting feeds. Go to Videos -> Importing feeds and create a new importing feed. In Feed URL set the first URL of your URLs list that you created when using feed builder. Choose KVS (XML) parser. In Duplicates prefix specify domain name of your primary project you export videos from. If you create multiple importing feeds for the same project, please make sure that all share that same duplicates prefix, so that's why the best idea is to use source project domain name here. Under Interval field choose Run this feed only once and deactivate. Since you have paginated data (if source site contains more than 10k videos you will most likely end up with feed data split into 10k-volume pages), you will need to import each page separately. The proposed way is to first configure feed to import all videos from the first page, then wait for it to be deactivated and re-configure to import from the second page URL, and etc. until you import all pages with all existing videos. After that you should change Feed URL to the last URL you created that will pull latest 1000 videos and configure its Interval to how often you want feed to pull new videos. Then feed will automatically pull new videos for the rest of time. Under Feed data check all data fields you want to import. In Video data section set Video status to Active, Video screenshots to Use screenshots from feed, Video publishing dates to Use publishing date from feed. Finally in Video adding mode option you should choose whether you want to embed, or hotlink, or download videos. If you want to download, then in most cases you will need to set All formats that have match by postfixes (without processing) in Format option. However in order this option to work correctly, you should go to Settings -> Video formats in this project and make sure the formats are configured in the same way they are configured on source site. At least the formats that you want to be downloaded via feeds. When we say "in the same way", we mean that their postfixes and statuses should be set accordingly. For example if on source project you have 480p, 720p formats and MP4 preview format, then you should make sure your importing site also have these formats with the same postfixes. If you don't want, for example, to download MP4 preview format and you want it to be created on the importing project with some other options - create it with another postfix syntax and required status. Then importing feed should download 480p and 720p files only and KVS will auto-create your custom MP4 preview from them. NOTE: for testing purposes it is recommended to set Max videos per execution to some small number (e.g. 10-100) for the first iteration to make sure you configured importing feed correctly. Then you can remove this limit and run feed again allowing it importing the whole set of data.
KVS admin panel supports all possible ways to add videos. All you need is to stick to the way which best suits your situation, whether you have existing content or not. There are different content types supported for videos (upload, embedding, hotlinking and other), you may also need to understand how they are different from each other in What video types are supported in KVS tube script and how they are different post. When using upload content type, it is also possible to upload files differently depending on whether you want them to be re-converted or not, please check Different ways to upload video files into KVS for more details. Adding videos manually This is the most obvious way to add videos. The form can be found in Videos -> Add video section of admin panel. You can specify all possible options there and send new video to conversion engine. You don't need to log in to website area as a specific user to add videos from this user - you can do that in admin panel by selecting the needed user in add video form. In Settings -> Content Settings section under Video adding / editing settings you can select some options that affect this form: the most valuable are default user and default status. Based on which video content type you want to add, KVS will show only fields that are related to the selected content type: Adding videos manually is very simple and straightforward process, but this won't work well if you need to add many videos. Therefore please check different mass importing options available in KVS. Mass importing videos from a text file or Excel spreadsheet This feature should be used in the following cases: You have a text file created by some other website or program with the needed list of videos to import (KVS export feature can also generate such text file to export videos from one KVS to another). You have a list of video files uploaded to your server and you want to import them into KVS with specifying all additional info. For accessing this functionality you should go to Videos -> Import videos. There you can copy-paste your text field into a data textarea, make sure to configure Field separator correctly (\t for copy-paste from Excel, semicolon (;) for CSV, or other separator if you are using plain text file). In most cases Line separator should not be changed. Then you need to select list of Fields in the exact order you have in your import data. IMPORTANT! Some CSV files may have separator at the end of every line, which actually means that there is one more empty field at the end of every line. In this case you should add one more additional field and select * Skip this field * option there. If you still have error indicating wrong number of fields in every single line, you should carefully check whether you've selected the correct separator and whether the number of fields matches the number of separators in the line + 1. The number of fields should be +1 all the time! Here are some examples. Single text file import with pipe (|) separator: Specifying data in Excel and then copy-paste: When using import functionality it is very important to understand which data you need to provide based on which video content type you are trying to import. Video upload: you should either provide video file URL in Source video file (with processing) field, or one / several Video file "XXX" (without processing) fields. Video hotlink: you should provide video file URL in Hosted file URL (hotlinking) field. Additionally you may specify Duration and Overview screenshots (sources comma-separated). If you don't provide duration or screenshots in import data, KVS will have to download the file you are hotlinking for detecting its duration and screenshots. This may significantly increase video processing time. Video embed: you should specify Embed code (embedded video) field. In addition if you also specify Hosted file URL (hotlinking) field, KVS will temporarily download file from this URL and will detect duration and screenshots automatically from it. Otherwise if you don't know video file URL or you don't want KVS to spend time on its download, you should specify Duration and Overview screenshots (sources comma-separated) fields. Pseudo video: you should specify Outgoing URL (pseudo video) field. In addition if you also specify Hosted file URL (hotlinking) field, KVS will temporarily download file from this URL and will detect duration and screenshots automatically from it. Otherwise if you don't know video file URL or you don't want KVS to spend time on its download, you should specify Duration and Overview screenshots (sources comma-separated) fields. Import feature supports the huge list of other options, but for novice users in most cases you don't need to touch them and better use everything by default. If you import many videos, you may need to look at post date randomization options, so that you can import videos to be published in some future or past period. For example you can import 1000 videos as they have been added one by one since 2010. Anyway, you can always do the same via massedit feature, so not something should be really important during import phase. After you submit the import, KVS will validate each single line and will show you summary so that you can confirm it. The validation may take much time depending on your input data. By default KVS validates all URLs to be valid, which may take some time. You can disable URL validation in import options. It is recommended to start with several lines first to make sure you understand the concept. When looking into import validation summary you will be able to see all errors and warnings. Warnings are not critical, they just indicate potential issues in your input data, which are not necessary real issues. Errors are critical and indicate the lines that can't be imported. If there are any lines without errors, you can still confirm import and lines which have errors will be simply skipped from being imported. If all lines have errors, you will not be able to confirm the import. After you confirm import summary, KVS will create a background task for import. You can see import tasks in Administration -> Imports log. There you can see data submitted with each import as well as import status and log. All imports are processed on background. You don't have to wait for anything here. Import tasks may have errors, even if validation didn't find any issues. There may be issues with downloading files, network issues and some other. IMPORTANT! If you submitted an import but you don't see any videos added, you may need to go to Administration -> Imports log and look at what is going on there. It may be that all import lines failed. Looking into import log will help you understand the issue. It may also be a case that your import is stuck due to lack of free space on your main server (since import in most cases download new files to your server there is a protection to check disk space, so that your HDD is not 100% filled). In this case import will automatically continue as soon as more free space is available. When you use this functionality really often, it may help to create an import pattern, so that you do not select the same options every time. You may define as much different import patterns as needed, then you just need to select the needed pattern and copy-paste import data. The rest of options will be defaulted from import pattern. Uploading video files via FTP and use FTP content uploader plugin If you already have a list of video files and want to quickly upload them into KVS, the easiest way is to use FTP content uploader plugin. You just need to create a folder somewhere under your www root and upload all your video files there. Then in FTP content uploader plugin you should specify full server path to that folder, or relative path under your www root (not starting with /, e.g. upload for http://domain.com/upload folder). Each video file in this folder will be treated as a standalone video and will be uploaded into KVS with just a few clicks: In advanced scenario here if you want to upload multiple formats for the same video as well as screenshots and / or description, you may need to create subfolders for each video. All files in a subfolder will be considered to belong to each video. The following rules will be applied: Subfolder name means title of a video. ZIP file means list of screenshots uploaded for a video. JPG file means main screenshot uploaded for a video. TXT file means description specified for a video. Video files in subfolder may be treated differently based on your selection in Video formats option: -- If you want to upload as a source file with processing, then the biggest video file will be chosen as a source file, other video files will be ignored. -- If you want to upload as format files without processing, the plugin will compare file names with video formats to find postfix matching. As a result, plugin will show you a summary of all detected files and after you confirm that, they will be uploaded as videos into KVS. It is recommended to enable delete option (if not a problem that files will be deleted), because this will use rename operation on filesystem, which works instantly in comparison to copy operation. Using this plugin will not allow you to specify other data, such as categories, tags and etc. It is expected that you want to create videos first and then you may manually edit them in admin panel, after they are imported. If you want to specify all this data at import time, you need to consider using import feature (2). Since your video files are already uploaded on your server, you just need to specify their URLs for the import to take them. At the same time you can also specify other data needed. Automating mass importing videos with a feed This feature should be used in the following cases: Import data is regularly updated and you want to automate importing the updates. You want to automatically sync content between 2 or more KVS projects. You have a software that uploads video files to your FTP and you want then to auto-import them (for example CCTV or webcams). To access this functionality go to Videos -> Add importing feed section. Feeds are very similar to mass import described in previous section, but the main purpose of a feed is to automate importing without your intervention. Every feed can be configured to query a text file from the given URL every given interval and add all new content that appears in this text file. Many options here are very similar to mass importing options: for plain text file format you need to specify field and line separators and select list of data columns. When a feed gets executed for the first time, it will add all available content from the given file if you don't specify any limit. Starting from the next execution feed will only add new content that appears in the text file. Here it is very important to understand how duplicates are being calculated. First of all you should specify Duplicates prefix - any unique string for this feed. Then if using CSV / plain text format you should also indicate the key field from this file - a field which is unique for every line. It can be video ID, or video URL or title - any unique field. Duplicates prefix is needed for cases when you need to separate content between different feeds. Imagine that you used video ID as a key field. If you import data from multiple providers by using multiple feeds, IDs may overlap there, e.g. video #1234 may exist at both provider 1 and provider 2. If KVS only used ID to identify duplicates, it would end up by considering video #1234 from provider 1 and video #1234 from provider 2 as duplicates, so one of these videos wouldn't get added. But if you specify Duplicates prefix unique for each feed - this will not be a problem anymore. Video #1234 from feed 1 will not be a duplicate of video #1234 from feed 2. At the same time, you may have several different feeds adding content from the same source (for example 1 feed is pulling all videos from Category 1 and another feed is pulling all videos from Category 2). Since these feeds pull content from the same source and the content may overlap, you should specify the same value for Duplicates prefix for both feeds. Because a single video #1234 may have both Category 1 and Category 2 and thus it will appear in both feeds. If you've specified different prefixes, this video will be added twice. But if prefixes are the same - videos #1234 in both feeds will be considered as duplicates. Grabbing videos from 3rd party sites KVS features extremely powerful tube grabbers plugin. With this plugin you can import tons of content, even if you don't have any. We will not speculate on whether it is ethic or not to grab content from 3rd party sites, you should decide this yourself. In order to use grabbers you first need to install and configure some. KVS provides a list of most popular tube grabbers and from time to time we expand it with new grabbers. There is also a default video grabber which can download videos from abstract tube sites. However specific grabbers are more powerful: They may support embed codes if source site provides them. They may support additional video data, such as categories, tags, models, rating, views, date, etc. They may support downloading video files in multiple qualities, so that you don't need to convert them on your server. Most of video grabbers require youtube-dl library to be installed on your server. If you don't have it, please ask host support to install it for you, otherwise it won't be possible to use many grabbers. After you install some grabbers, you should configure them. You should select Mode option to indicate whether you want this grabber to download, embed or add videos as pseudo videos. Then you also need to choose which data fields you want to grab; the set of fields is different for each grabber and is based on which data can be parsed from the grabbed site. Finally there are filtering options, that help you to maintain grabbed content quality. IMPORTANT! when you want to use grabbers to download videos to your server, like with any other methods you can decide whether you want the downloaded video files to be processed, or you want them to be saved without processing. Here you can use 2 different tactics based on whether your project has enough computing power or not: If your project doesn't have much computing power, it makes sense to download all files without processing. In order to configure this you should set Multiple value for Quality option and then based on the number of video files provided by grabber match your video formats: If your project has enough computing power, it may be faster for you to download only 1 single file from grabber and have other files created by KVS. This may work faster, because download from grabbers is typically slow due to speed throttling by the source site. In this case you should select the needed file under Quality option and assign it to the correct format; so that all other formats will be created by KVS: NOTE: in some cases if you want to process all files, even the downloaded one, you will need to set upload as Source file. One important issue you'll find when using grabbers is how to manage your categorization efficiently. Different sites may have many similar categories named differently, so that grabbers may create too many categories in your database. There are several workarounds to deal with this described in Categorization best practices article. There are 2 ways you can import videos from grabbers plugin: you can import videos manually or you can configure grabbers to ping video lists from time to time and add videos from these lists. When using grabbers for the first time we recommend running manual imports first to understand how grabbers work and whether the grabbed videos are in the way you want them to be added. In order to import videos manually you should go to Upload content using grabbers manually section in grabbers plugin. There you need to specify list of URLs you want to grab. In this list you can specify both individual videos (if you want to pick each video yourself) and also you can specify video list URLs or video RSS URLs to grab the whole lists. Here is where you can feel the real power of using site-specific grabbers - you can use your browser to navigate source site and find a page which lists the needed videos - this can be a page listing all top rated videos, or most viewed videos for the last week, or videos from a specific category, or videos from a specific user and etc. You can then copy-paste URL you see in browser address field and grabber will open that page and will add all videos from it. Many tube grabbers also support pagination mode, which means that grabber will be able not only to add all videos you see on that URL, but also to navigate to page #2, to page #3 and etc. of a paginated list and find all videos there as well. In order to enable that, you need to specify a number of videos you want to grab right after the URL separated by pipe symbol, e.g.: https://www.kvs-demo.com/top-rated/|1000 This command tells KVS grabber to import 1000 videos from this page. If this page is a paginated list and shows only 50 videos per page, and the grabber supports pagination, KVS will query 20 more pages from this list to add 1000 videos in total. Using this approach you can mass import 100 top rated videos from every category on the site by simply specifying URLs to category pages and number of videos you want to grab from them. If you do not specify a number after list URL, KVS will add only those videos that are displayed initially on this page, e.g. only 50 videos for this sample top videos list: https://www.kvs-demo.com/top-rated/ Finally, when you have imported sample list of videos and you are happy with them, you can move forward and configure auto-pilot. Auto-pilot is exactly the same as importing manually via specifying list URLs and counts. The only difference is that auto-pilot will do that automatically for you, e.g. it will query the specified pages every N hours as you configure and will check if there are any new videos there. Therefore if you want it to import new videos every time, you should make sure to provide list URLs, which are sorted by new videos first. If you recall our previous example the top.html will not work here, since top lists are rarely updated. Instead in auto-pilot settings you should specify URLs which are updated every N hours you plan to query for new videos, for example latest videos page: https://www.kvs-demo.com/latest-updates/|100 By setting the above URL to auto-pilot you will require KVS to grab new videos from this website every N hours. Then you simply do not need to upload any videos manually, because they will be added and added every N hours you configure (if the source site is updated of cause with the new videos). If you want to grab only from specific categories, or tags, or models or whatever, you will need to specify multiple URLs, each for the list you want to grab, e.g. like this for grabbing from 3 categories: https://www.kvs-demo.com/categories/rock-music/|100 https://www.kvs-demo.com/categories/house-music/|100 https://www.kvs-demo.com/categories/pop-music/|100 While you can also use RSS with grabbers, using the URLs gives you much more power than RSS. In most cases you can adjust sorting or even do some filtering in the URL, so that grabbers only import content you'd like to appear on your site. Here are some important highlights with using tube grabbers in KVS: Use specific grabbers designed for specific sites, they support more options. If no specific grabber is available for the site you want to parse, you can try using default video grabber if it suits you (this grabber only supports download mode). But this grabber won't support 100% sites, it can only grab from sites that do have video URLs clearly printed in HTML code. When downloading from other sites, your IP may be blacklisted for frequent downloads. KVS has no control on that other than letting you specify timeout to avoid frequent requests. Some grabbers support proxies that can be listed in their settings. Try importing manually first before you configure auto-pilot. If you want to grab initial set of content, grab it manually and use auto-pilot for pulling new videos. Use web URLs instead of RSS to download from video lists. Make sure videos in the list are filtered and sorted in the way you want to grab them. You can still use RSS if you want to. Default grabber only supports RSS lists. Grabbers may become broken from time to time if source sites gets updated. If you find this issue, please report to KVS support and we will update the broken grabber to be working again. With KVS you can create your own grabbers and use them on your projects. You can find more details in How to create custom tube video grabber for KVS article. Combining grabbers and mass import Grabbers plugin can import many videos from other sites, but sometimes you may need more flexible way to grab video files and at the same time specify your own title, description, categorization and so on. In this case you can combine grabbers functionality to download files and mass import flexibility to specify data. However in this case you need to manually select videos and specify their direct page URLs, grabbing multiple videos from lists will not be possible. In order to use grabbers in mass import you should specify video page URL in Video page URL (grabbers) field. Then you can also specify any additional data you need. The key point here is that your custom fields specified in import data will override fields from grabbers. For example you can specify title and categories, so these fields will be taken from import. All other data will be taken from grabber. Using this approach will take much of your time to prepare data, but you will also benefit from more accurate and unique SEO, which is best for your project.
KVS supports different content types for videos (uploading, embedding, hotlinking and other), here is full information about them: What video types are supported in KVS tube script and how they are different. When using upload content type, you upload a source video file into KVS and then let KVS process it and store it using KVS storage system. During video processing phase KVS will create all required video formats defined in your settings. KVS will also try to create all conditionally-optional video formats, if the uploaded source file meets their size and duration limits. The most common example here is when you have multiple video formats defined for multiple qualities, e.g. 360p, 480p and 720p. In such scenario 360p should be configured as required format, 480p and 720p should be configured as conditionally-optional formats. So that 480p and 720p are only created for source video files that allow resizing them to 480p or 720p respectfully. For example if you upload source file in 720p - all 3 formats will be created. If you upload source file in 480p - only 360p and 480p will be created, 720p format will be skipped. For source files with 360p or less quality KVS will only create 360p format, because this is the only required format in this configuration. When using the described approach you don't need to think about source file original quality, all you need is to upload it as a source file with processing and KVS will then create all necessary formats. This processing will however take much time, because operation to create one video file from another uses your server resources very intensively. There can be scenarios when you will want to skip this processing: You don't need to have multiple video formats, you need to show video in only one quality. You already have video files in the needed format(s), for example you already prepared them using desktop software or you import existing video files from external sources. You don't need any resize or watermarking on video files. KVS provides ability to skip video processing for all or some files. In order to do that you should upload your video files not as source files, but directly into the needed video format(s). When a file is uploaded as a video format, KVS will not do any re-conversion with this file, the file will be saved in the same form as you've uploaded it. By doing this you can significantly decrease video processing time, but there are certain limitations you should be aware of: No watermarking will be applied to the uploaded file as well as no resize and duration trim. You should make sure that the uploaded file has valid format (e.g. if you upload it into MP4 format, the file should be a valid MP4 file, not any other video file). If you try to upload an invalid file, most probably there will be an error thrown during processing phase, however this depends on your ffmpeg version and it is also possible that there will be no error and you will end up with a video that is not playable via web player even if you can play it with your desktop. If you have multiple video formats, you can choose to upload them all and fully skip conversion, or upload only one of them and let KVS create the rest. Depending on which functionality you typically use to upload videos into KVS, they all provide ability to upload in both ways. Naming convention for this option is as following: Source file (with processing): means that the uploaded video file will be considered as a source for all video formats that will be created by KVS. Format "XXX" (without processing): means that the uploaded video file will be uploaded directly as the given format, so it won't be processed by KVS (other formats may be created from it if required). Manual video adding When adding videos manually you can see a field to upload as a source file (Source file) and separate fields to upload into each specific video format (e.g. Format "XXX"): Videos mass import In mass import for each field you can either select if it should be considered as a source file (Source video file (with processing)), or as a specific video format (Video file "XXX" (without processing)): Videos importing feeds In importing feed settings you can choose whether all video files from this feed should be considered as source files (Source file (with processing)), or as files of a specific format (Format "XXX" (without processing)). Importing feeds do not support uploading multiple files in multiple video formats: FTP content uploader plugin Using FTP content uploader plugin is quite tricky if you have multiple video formats and want to upload them as formats without processing. In order to do that you should name your video files based on postfixes you have in video formats. For example if you want to upload only 1 format per each video, you can still put all video files into 1 upload folder, but make sure that their file extension match the postfix of the video format you want it to be uploaded to, e.g. test.mp4 will match video format with .mp4 postfix. Also you should set Video formats option to detect postfixes: If you want to upload multiple video formats for each video, then you should put files for the same video into a subfolder under your upload folder. Each file should have filename based on video format postfix you want it to be uploaded to, e.g. test.mp4, test_480p.mp4 and test_720p.mp4 for formats with .mp4, _480p.mp4 and _720p.mp4 postfixes. Then on summary page you can check if all the formats have been detected as expected: Grabbers plugin In grabbers plugin if you want to download only 1 video file, you can simply select whether it should be considered as a source file or specific format file under upload as option: If the source site provides multiple video formats, you can download as many of them as needed and upload as specific video formats. In order to get these fields displayed you should switch Quality field to Multiple value: NOTE: downloading multiple files from grabber could be slowly due to bandwidth throttling on the source site. If your project's conversion power is high, it may be faster to download only 1 file and then create other formats via KVS. Summary Uploading as a source file with processing: Should be used by inexperienced users. You can upload any source file with any quality (480p, 720p and etc), KVS will automatically decide which formats to create. KVS will create all the needed formats with all their settings, e.g. watermarks, quality, duration trim. Works slow due to video conversion utilized. Uploading as a format file without processing: Should be used by experienced users if all restrictions are not a problem (you don't need watermark and etc. see below). Video file format and quality (480p, 720p and etc.) should exact match video format settings. No watermarking, quality change and duration trim will be done. Works incredibly faster.