Jump to content

Search the Community

Showing results for tags 'howto'.

  • Search By Tags

    Type tags separated by commas.
  • Search By Author

Content Type


Forums

  • General Discussion
    • Announcements
    • Business Discussion
    • Business Services
    • Sell and Buy
  • Themes & templates
    • KVS default theme
    • KVS tablet theme
    • KVS paysite theme
    • KVS basic theme 1
    • KVS basic theme 2
    • KVS basic theme 3
  • Educational / support
    • Technical Support
    • FAQ
    • Educational Series

Find results in...

Find results that contain...


Date Created

  • Start

    End


Last Updated

  • Start

    End


Filter by number of...

Joined

  • Start

    End


Group


About Me

  1. KVS provides API to use youtube-dl server library for scrapping videos from other tube sites. You can implement your own grabber class in PHP language and upload it into KVS. Here is how this can be done. The example features fully working custom youtube grabber (KVS has built-in grabber for youtube by the way). NOTE: it is not strictly required to use youtube-dl API, it is also possible to create a completely custom grabber with your own code. Implementing grabber class using youtube-dl API Create CustomGrabberYoutube.php with the following code (also attached here as a text file): <?php // when you change classname, change it at the very bottom as well in this line: // $grabber = new CustomGrabberYoutube(); class CustomGrabberYoutube extends KvsGrabberVideoYDL { // =============================================================================================================== // infrastructure methods // =============================================================================================================== public function get_grabber_id() { //prefix your grabber ID with "custom_" return "custom_videos_youtube"; } public function get_grabber_name() { // name displayed in admin panel return "youtube.com"; } public function get_grabber_version() { // this is required for grabbers that are autoupdated from KVS return "1"; } public function get_grabber_domain() { // domain name, KVS will check this to find out if this grabber is suitable for the given URL return "youtube.com"; } public function get_supported_url_patterns() { // returns list of regexp patterns that describe video URLs, for youtube this pattern will match // https://www.youtube.com/watch?v=htOroIbxiFY return array("/https?:\/\/(www\.)?youtube\.com\/watch.*/i"); } public function can_grab_description() { // return true if your grabber is going to provide description for each video return false; } public function can_grab_categories() { // return true if your grabber is going to provide categories for each video return false; } public function can_grab_tags() { // return true if your grabber is going to provide tags for each video return false; } public function can_grab_models() { // return true if your grabber is going to provide models for each video return false; } public function can_grab_content_source() { // return true if your grabber is going to provide content source for each video return false; } public function can_grab_date() { // return true if your grabber is going to provide date for each video return false; } public function can_grab_rating() { // return true if your grabber is going to provide rating for each video return false; } public function can_grab_views() { // return true if your grabber is going to provide views for each video return false; } public function can_grab_video_files() { // this should be true for youtube-dl return true; } public function get_supported_qualities() { // list of supported video qualities, should match what youtube-dl returns in its info under formats // run this command: // youtube-dl --dump-json https://www.youtube.com/watch?v=PhDXRCLsqz4 >> test.json // and open test.json in Firefox, find "formats" array and look into the available formats // youtube has too many formats, KVS only supports formats with "ext"="mp4" // you can list them here and you will be able to select from them in grabber settings return array('360p', '720p'); } public function get_downloadable_video_format() { // for youtube-dl grabber KVS only supports mp4 formats return 'mp4'; } public function can_grab_lists() { // return true if you want to allow this grabber to grab lists and thus be used on autopilot // if true, you will also need to implement grab_list() method - see below return false; } // =============================================================================================================== // parsing methods - modify if you need to parse lists or add additional info // =============================================================================================================== public function grab_list($list_url, $limit) { // this method is used to grab lists of videos from the given list URL // $limit parameter means the number of videos to grab (including pagination) // if $limit == 0, then you just need to find all videos on the given URL, no need to care about pagination $result = new KvsGrabberListResult(); // $page_content here is the HTML code of the given page $page_content = $this->load_page($list_url); // parse $page_content and add all video URLs to the result // consider pagination if needed // you can use $this->load_page($list_url) method to get HTML from any URL $result->add_content_page("https://youtube.com/video1"); $result->add_content_page("https://youtube.com/video2"); $result->add_content_page("https://youtube.com/video3"); return $result; } protected function grab_video_data_impl($page_url, $tmp_dir) { // by default the base class will populate these fields (if provided by youtube-dl): // - title // - MP4 video files for the qualities listed in get_supported_qualities() function // - description (should be enabled in can_grab_description() function) // - date (should be enabled in can_grab_date() function) // - tags (should be enabled in can_grab_tags() function) // - categories (should be enabled in can_grab_categories() function) $result = parent::grab_video_data_impl($page_url, $tmp_dir); if ($result->get_error_code() > 0) { return $result; } // do any custom grabbing here for additional fields, which are not supported by youtube-dl // $page_content here is the HTML code of the given video page //$page_content = $this->load_page($page_url); // parse HTML code and set additional data into $result, e.g. data which is not provided by youtube-dl //$result->set_rating(85); //$result->set_votes(10); //$result->set_views(123874); //$result->set_content_source("Content Source Name"); //$result->add_model("Model 1"); //$result->add_model("Model 2"); return $result; } } $grabber = new CustomGrabberYoutube(); KvsGrabberFactory::register_grabber_class(get_class($grabber)); return $grabber; The code has comments where needed. Basically youtube-dl provides main video info, such as title, description, tags, categories, date and files. If this is enough for you, you should only modify set of methods on top grouped under infrastructure methods section. These methods are designed to integrate grabber into KVS, so you should change them as described. You should also modify grabber class name in 2 places (top and bottom) and make sure that grabber class name is unique and has Custom in its name (to avoid intersections with any future grabbers we will add). If you want to implement parsing lists or add additional info, you should modify parsing methods as explained in the code. Implementing grabber class without youtube-dl Here is example grabber class that is not using youtube-dl. Put your custom parsing logic: <?php // when you change classname, change it at the very bottom as well in this line: // $grabber = new CustomGrabberYoutube(); class CustomGrabberYoutube extends KvsGrabberVideo { // =============================================================================================================== // infrastructure methods // =============================================================================================================== public function get_grabber_id() { //prefix your grabber ID with "custom_" return "custom_videos_youtube"; } public function get_grabber_name() { // name displayed in admin panel return "youtube.com"; } public function get_grabber_version() { // this is required for grabbers that are autoupdated from KVS return "1"; } public function get_grabber_domain() { // domain name, KVS will check this to find out if this grabber is suitable for the given URL return "youtube.com"; } public function get_supported_url_patterns() { // returns list of regexp patterns that describe video URLs, for youtube this pattern will match // https://www.youtube.com/watch?v=htOroIbxiFY return array("/https?:\/\/(www\.)?youtube\.com\/watch.*/i"); } public function can_grab_description() { // return true if your grabber is going to provide description for each video return true; } public function can_grab_categories() { // return true if your grabber is going to provide categories for each video return true; } public function can_grab_tags() { // return true if your grabber is going to provide tags for each video return true; } public function can_grab_models() { // return true if your grabber is going to provide models for each video return true; } public function can_grab_content_source() { // return true if your grabber is going to provide content source for each video return true; } public function can_grab_date() { // return true if your grabber is going to provide date for each video return true; } public function can_grab_rating() { // return true if your grabber is going to provide rating for each video return true; } public function can_grab_views() { // return true if your grabber is going to provide views for each video return true; } public function can_grab_video_files() { // return true if your grabber is going to provide video files for each video return true; } public function can_grab_video_embed() { // return true if your grabber is going to provide embed code for each video return true; } public function can_grab_video_duration() { // return true if your grabber is going to provide duration for each video return true; } public function can_grab_video_screenshot() { // return true if your grabber is going to provide screenshot for each video return true; } public function get_supported_qualities() { // list of supported video qualities that your grabber provides return array('360p', '720p'); } public function get_downloadable_video_format() { // only grabbers that return MP4 files are supported return 'mp4'; } public function can_grab_lists() { // return true if you want to allow this grabber to grab lists and thus be used on autopilot // if true, you will also need to implement grab_list() method - see below return false; } // =============================================================================================================== // parsing methods // =============================================================================================================== public function grab_list($list_url, $limit) { // this method is used to grab lists of videos from the given list URL // $limit parameter means the number of videos to grab (including pagination) // if $limit == 0, then you just need to find all videos on the given URL, no need to care about pagination $result = new KvsGrabberListResult(); // $page_content here is the HTML code of the given page $page_content = $this->load_page($list_url); // parse $page_content and add all video URLs to the result // consider pagination if needed // you can use $this->load_page($list_url) method to get HTML from any URL $result->add_content_page("https://youtube.com/video1"); $result->add_content_page("https://youtube.com/video2"); $result->add_content_page("https://youtube.com/video3"); return $result; } protected function grab_video_data_impl($page_url, $tmp_dir) { $result = new KvsGrabberVideoInfo(); // $page_code here is the HTML code of the given video page $page_code = $this->load_page($page_url); if (!$page_code) { $result->log_error(KvsGrabberVideoInfo::ERROR_CODE_PAGE_UNAVAILABLE, "Page can't be loaded: $page_url"); return $result; } // parse HTML code and set data into $result // replace with your parsing logic $result->set_canonical($page_url); $result->set_title("Demo title"); $result->set_description("Demo description long description long description long description long description."); $result->set_screenshot("http://www.localhost.com/test/test.jpg"); $result->set_duration(30); $result->set_date(time()); $result->set_views(1526); $result->set_rating(87); $result->set_votes(11); $result->set_embed("<div>embed code</div>"); $result->add_category("Category 1"); $result->add_category("Category 2"); $result->add_category("Category 3"); $result->add_tag("Tag 1"); $result->add_tag("Tag 2"); $result->add_tag("Tag 3"); $result->add_model("Model 1"); $result->add_model("Model 2"); $result->add_model("Model 3"); $result->set_content_source("Content Source 1"); $result->add_video_file("360p", "http://www.localhost.com/test/test_360p.mp4"); $result->add_video_file("720p", "http://www.localhost.com/test/test_720p.mp4"); $result->add_custom_field(1, "Custom1"); $result->add_custom_field(3, "Custom3"); return $result; } } $grabber = new CustomGrabberYoutube(); KvsGrabberFactory::register_grabber_class(get_class($grabber)); return $grabber; Testing grabber class Put grabber class file to your project root folder. Also create test_grabber.php file in the same folder with the following code: <?php header('Content-Type: text/plain; charset=utf8'); ini_set('display_errors', 1); error_reporting(E_ERROR | E_PARSE | E_COMPILE_ERROR); require_once('admin/plugins/grabbers/classes/KvsGrabber.php'); $grabber = require_once('CustomGrabberYoutube.php'); $grabber->init(new KvsGrabberSettings(), ""); if ($grabber instanceof KvsGrabberVideoYDL) { $grabber->set_ydl_binary('/usr/local/bin/youtube-dl'); } print_r($grabber->grab_video_data('https://www.youtube.com/watch?v=htOroIbxiFY', 'tmp')); Modify this code to your class name and specify your demo URL. Then run via browser: http://domain.com/test_grabber.php If everything is fine, you should see dumped info from the scrapped video. Installing grabber into KVS Just go to Plugins -> Grabbers in admin panel and upload your grabber class into Custom grabber field. Then after saving the form you will see your grabber installed marked with red color. You need to open this grabber settings and select Content mode = Download. Also enable the needed fields under Data. NOTE: If you don't see any fields under Data, then your grabber class doesn't return true from can_grab_xxx() methods. If you want to update grabber class, simply upload it again. It is recommended to increment version in get_grabber_version() method to stay sure on which version KVS is using. Finding the list of supported video files to grab If you don't know which formats source site provides (usually a subset of: 240p, 360p, 480p, 720p, 1080p), you can check that from youtube-dl: youtube-dl --dump-json https://www.youtube.com/watch?v=PhDXRCLsqz4 >> test.json This should generate test.json file which can be open in firefox to show JSON structure. Find a node called formats, it should be a list with items describing each supported format. KVS can only import formats with ext = mp4, you can list them in get_supported_qualities() method using XXXp notation, e.g. 360p, 720p. Here is sample screenshot for youtube: CustomGrabberYoutube.txt
  2. KVS admin panel supports all possible ways to add videos. All you need is to stick to the way which best suits your situation, whether you have existing content or not. There are different content types supported for videos (upload, embedding, hotlinking and other), you may also need to understand how they are different from each other in What video types are supported in KVS tube script and how they are different post. When using upload content type, it is also possible to upload files differently depending on whether you want them to be re-converted or not, please check Different ways to upload video files into KVS for more details. Adding videos manually This is the most obvious way to add videos. The form can be found in Videos -> Add video section of admin panel. You can specify all possible options there and send new video to conversion engine. You don't need to log in to website area as a specific user to add videos from this user - you can do that in admin panel by selecting the needed user in add video form. In Settings -> Content Settings section under Video adding / editing settings you can select some options that affect this form: the most valuable are default user and default status. Based on which video content type you want to add, KVS will show only fields that are related to the selected content type: Adding videos manually is very simple and straightforward process, but this won't work well if you need to add many videos. Therefore please check different mass importing options available in KVS. Mass importing videos from a text file or Excel spreadsheet This feature should be used in the following cases: You have a text file created by some other website or program with the needed list of videos to import (KVS export feature can also generate such text file to export videos from one KVS to another). You have a list of video files uploaded to your server and you want to import them into KVS with specifying all additional info. For accessing this functionality you should go to Videos -> Import videos. There you can copy-paste your text field into a data textarea, make sure to configure Field separator correctly (\t for copy-paste from Excel, semicolon (;) for CSV, or other separator if you are using plain text file). In most cases Line separator should not be changed. Then you need to select list of Fields in the exact order you have in your import data. IMPORTANT! Some CSV files may have separator at the end of every line, which actually means that there is one more empty field at the end of every line. In this case you should add one more additional field and select * Skip this field * option there. If you still have error indicating wrong number of fields in every single line, you should carefully check whether you've selected the correct separator and whether the number of fields matches the number of separators in the line + 1. The number of fields should be +1 all the time! Here are some examples. Single text file import with pipe (|) separator: Specifying data in Excel and then copy-paste: When using import functionality it is very important to understand which data you need to provide based on which video content type you are trying to import. Video upload: you should either provide video file URL in Source video file (with processing) field, or one / several Video file "XXX" (without processing) fields. Video hotlink: you should provide video file URL in Hosted file URL (hotlinking) field. Additionally you may specify Duration and Overview screenshots (sources comma-separated). If you don't provide duration or screenshots in import data, KVS will have to download the file you are hotlinking for detecting its duration and screenshots. This may significantly increase video processing time. Video embed: you should specify Embed code (embedded video) field. In addition if you also specify Hosted file URL (hotlinking) field, KVS will temporarily download file from this URL and will detect duration and screenshots automatically from it. Otherwise if you don't know video file URL or you don't want KVS to spend time on its download, you should specify Duration and Overview screenshots (sources comma-separated) fields. Pseudo video: you should specify Outgoing URL (pseudo video) field. In addition if you also specify Hosted file URL (hotlinking) field, KVS will temporarily download file from this URL and will detect duration and screenshots automatically from it. Otherwise if you don't know video file URL or you don't want KVS to spend time on its download, you should specify Duration and Overview screenshots (sources comma-separated) fields. Import feature supports the huge list of other options, but for novice users in most cases you don't need to touch them and better use everything by default. If you import many videos, you may need to look at post date randomization options, so that you can import videos to be published in some future or past period. For example you can import 1000 videos as they have been added one by one since 2010. Anyway, you can always do the same via massedit feature, so not something should be really important during import phase. After you submit the import, KVS will validate each single line and will show you summary so that you can confirm it. The validation may take much time depending on your input data. By default KVS validates all URLs to be valid, which may take some time. You can disable URL validation in import options. It is recommended to start with several lines first to make sure you understand the concept. When looking into import validation summary you will be able to see all errors and warnings. Warnings are not critical, they just indicate potential issues in your input data, which are not necessary real issues. Errors are critical and indicate the lines that can't be imported. If there are any lines without errors, you can still confirm import and lines which have errors will be simply skipped from being imported. If all lines have errors, you will not be able to confirm the import. After you confirm import summary, KVS will create a background task for import. You can see import tasks in Administration -> Imports log. There you can see data submitted with each import as well as import status and log. All imports are processed on background. You don't have to wait for anything here. Import tasks may have errors, even if validation didn't find any issues. There may be issues with downloading files, network issues and some other. IMPORTANT! If you submitted an import but you don't see any videos added, you may need to go to Administration -> Imports log and look at what is going on there. It may be that all import lines failed. Looking into import log will help you understand the issue. It may also be a case that your import is stuck due to lack of free space on your main server (since import in most cases download new files to your server there is a protection to check disk space, so that your HDD is not 100% filled). In this case import will automatically continue as soon as more free space is available. When you use this functionality really often, it may help to create an import pattern, so that you do not select the same options every time. You may define as much different import patterns as needed, then you just need to select the needed pattern and copy-paste import data. The rest of options will be defaulted from import pattern. Uploading video files via FTP and use FTP content uploader plugin If you already have a list of video files and want to quickly upload them into KVS, the easiest way is to use FTP content uploader plugin. You just need to create a folder somewhere under your www root and upload all your video files there. Then in FTP content uploader plugin you should specify full server path to that folder, or relative path under your www root (not starting with /, e.g. upload for http://domain.com/upload folder). Each video file in this folder will be treated as a standalone video and will be uploaded into KVS with just a few clicks: In advanced scenario here if you want to upload multiple formats for the same video as well as screenshots and / or description, you may need to create subfolders for each video. All files in a subfolder will be considered to belong to each video. The following rules will be applied: Subfolder name means title of a video. ZIP file means list of screenshots uploaded for a video. JPG file means main screenshot uploaded for a video. TXT file means description specified for a video. Video files in subfolder may be treated differently based on your selection in Video formats option: -- If you want to upload as a source file with processing, then the biggest video file will be chosen as a source file, other video files will be ignored. -- If you want to upload as format files without processing, the plugin will compare file names with video formats to find postfix matching. As a result, plugin will show you a summary of all detected files and after you confirm that, they will be uploaded as videos into KVS. It is recommended to enable delete option (if not a problem that files will be deleted), because this will use rename operation on filesystem, which works instantly in comparison to copy operation. Using this plugin will not allow you to specify other data, such as categories, tags and etc. It is expected that you want to create videos first and then you may manually edit them in admin panel, after they are imported. If you want to specify all this data at import time, you need to consider using import feature (2). Since your video files are already uploaded on your server, you just need to specify their URLs for the import to take them. At the same time you can also specify other data needed. Automating mass importing videos with a feed This feature should be used in the following cases: Import data is regularly updated and you want to automate importing the updates. You want to automatically sync content between 2 or more KVS projects. You have a software that uploads video files to your FTP and you want then to auto-import them (for example CCTV or webcams). To access this functionality go to Videos -> Add importing feed section. Feeds are very similar to mass import described in previous section, but the main purpose of a feed is to automate importing without your intervention. Every feed can be configured to query a text file from the given URL every given interval and add all new content that appears in this text file. Many options here are very similar to mass importing options: for plain text file format you need to specify field and line separators and select list of data columns. When a feed gets executed for the first time, it will add all available content from the given file if you don't specify any limit. Starting from the next execution feed will only add new content that appears in the text file. Here it is very important to understand how duplicates are being calculated. First of all you should specify Duplicates prefix - any unique string for this feed. Then if using CSV / plain text format you should also indicate the key field from this file - a field which is unique for every line. It can be video ID, or video URL or title - any unique field. Duplicates prefix is needed for cases when you need to separate content between different feeds. Imagine that you used video ID as a key field. If you import data from multiple providers by using multiple feeds, IDs may overlap there, e.g. video #1234 may exist at both provider 1 and provider 2. If KVS only used ID to identify duplicates, it would end up by considering video #1234 from provider 1 and video #1234 from provider 2 as duplicates, so one of these videos wouldn't get added. But if you specify Duplicates prefix unique for each feed - this will not be a problem anymore. Video #1234 from feed 1 will not be a duplicate of video #1234 from feed 2. At the same time, you may have several different feeds adding content from the same source (for example 1 feed is pulling all videos from Category 1 and another feed is pulling all videos from Category 2). Since these feeds pull content from the same source and the content may overlap, you should specify the same value for Duplicates prefix for both feeds. Because a single video #1234 may have both Category 1 and Category 2 and thus it will appear in both feeds. If you've specified different prefixes, this video will be added twice. But if prefixes are the same - videos #1234 in both feeds will be considered as duplicates. Grabbing videos from 3rd party sites KVS features extremely powerful tube grabbers plugin. With this plugin you can import tons of content, even if you don't have any. We will not speculate on whether it is ethic or not to grab content from 3rd party sites, you should decide this yourself. In order to use grabbers you first need to install and configure some. KVS provides a list of most popular tube grabbers and from time to time we expand it with new grabbers. There is also a default video grabber which can download videos from abstract tube sites. However specific grabbers are more powerful: They may support embed codes if source site provides them. They may support additional video data, such as categories, tags, models, rating, views, date, etc. They may support downloading video files in multiple qualities, so that you don't need to convert them on your server. Most of video grabbers require youtube-dl library to be installed on your server. If you don't have it, please ask host support to install it for you, otherwise it won't be possible to use many grabbers. After you install some grabbers, you should configure them. You should select Mode option to indicate whether you want this grabber to download, embed or add videos as pseudo videos. Then you also need to choose which data fields you want to grab; the set of fields is different for each grabber and is based on which data can be parsed from the grabbed site. Finally there are filtering options, that help you to maintain grabbed content quality. IMPORTANT! when you want to use grabbers to download videos to your server, like with any other methods you can decide whether you want the downloaded video files to be processed, or you want them to be saved without processing. Here you can use 2 different tactics based on whether your project has enough computing power or not: If your project doesn't have much computing power, it makes sense to download all files without processing. In order to configure this you should set Multiple value for Quality option and then based on the number of video files provided by grabber match your video formats: If your project has enough computing power, it may be faster for you to download only 1 single file from grabber and have other files created by KVS. This may work faster, because download from grabbers is typically slow due to speed throttling by the source site. In this case you should select the needed file under Quality option and assign it to the correct format; so that all other formats will be created by KVS: NOTE: in some cases if you want to process all files, even the downloaded one, you will need to set upload as Source file. One important issue you'll find when using grabbers is how to manage your categorization efficiently. Different sites may have many similar categories named differently, so that grabbers may create too many categories in your database. There are several workarounds to deal with this described in Categorization best practices article. There are 2 ways you can import videos from grabbers plugin: you can import videos manually or you can configure grabbers to ping video lists from time to time and add videos from these lists. When using grabbers for the first time we recommend running manual imports first to understand how grabbers work and whether the grabbed videos are in the way you want them to be added. In order to import videos manually you should go to Upload content using grabbers manually section in grabbers plugin. There you need to specify list of URLs you want to grab. In this list you can specify both individual videos (if you want to pick each video yourself) and also you can specify video list URLs or video RSS URLs to grab the whole lists. Here is where you can feel the real power of using site-specific grabbers - you can use your browser to navigate source site and find a page which lists the needed videos - this can be a page listing all top rated videos, or most viewed videos for the last week, or videos from a specific category, or videos from a specific user and etc. You can then copy-paste URL you see in browser address field and grabber will open that page and will add all videos from it. Many tube grabbers also support pagination mode, which means that grabber will be able not only to add all videos you see on that URL, but also to navigate to page #2, to page #3 and etc. of a paginated list and find all videos there as well. In order to enable that, you need to specify a number of videos you want to grab right after the URL separated by pipe symbol, e.g.: https://www.kvs-demo.com/top-rated/|1000 This command tells KVS grabber to import 1000 videos from this page. If this page is a paginated list and shows only 50 videos per page, and the grabber supports pagination, KVS will query 20 more pages from this list to add 1000 videos in total. Using this approach you can mass import 100 top rated videos from every category on the site by simply specifying URLs to category pages and number of videos you want to grab from them. If you do not specify a number after list URL, KVS will add only those videos that are displayed initially on this page, e.g. only 50 videos for this sample top videos list: https://www.kvs-demo.com/top-rated/ Finally, when you have imported sample list of videos and you are happy with them, you can move forward and configure auto-pilot. Auto-pilot is exactly the same as importing manually via specifying list URLs and counts. The only difference is that auto-pilot will do that automatically for you, e.g. it will query the specified pages every N hours as you configure and will check if there are any new videos there. Therefore if you want it to import new videos every time, you should make sure to provide list URLs, which are sorted by new videos first. If you recall our previous example the top.html will not work here, since top lists are rarely updated. Instead in auto-pilot settings you should specify URLs which are updated every N hours you plan to query for new videos, for example latest videos page: https://www.kvs-demo.com/latest-updates/|100 By setting the above URL to auto-pilot you will require KVS to grab new videos from this website every N hours. Then you simply do not need to upload any videos manually, because they will be added and added every N hours you configure (if the source site is updated of cause with the new videos). If you want to grab only from specific categories, or tags, or models or whatever, you will need to specify multiple URLs, each for the list you want to grab, e.g. like this for grabbing from 3 categories: https://www.kvs-demo.com/categories/rock-music/|100 https://www.kvs-demo.com/categories/house-music/|100 https://www.kvs-demo.com/categories/pop-music/|100 While you can also use RSS with grabbers, using the URLs gives you much more power than RSS. In most cases you can adjust sorting or even do some filtering in the URL, so that grabbers only import content you'd like to appear on your site. Here are some important highlights with using tube grabbers in KVS: Use specific grabbers designed for specific sites, they support more options. If no specific grabber is available for the site you want to parse, you can try using default video grabber if it suits you (this grabber only supports download mode). But this grabber won't support 100% sites, it can only grab from sites that do have video URLs clearly printed in HTML code. When downloading from other sites, your IP may be blacklisted for frequent downloads. KVS has no control on that other than letting you specify timeout to avoid frequent requests. Some grabbers support proxies that can be listed in their settings. Try importing manually first before you configure auto-pilot. If you want to grab initial set of content, grab it manually and use auto-pilot for pulling new videos. Use web URLs instead of RSS to download from video lists. Make sure videos in the list are filtered and sorted in the way you want to grab them. You can still use RSS if you want to. Default grabber only supports RSS lists. Grabbers may become broken from time to time if source sites gets updated. If you find this issue, please report to KVS support and we will update the broken grabber to be working again. With KVS you can create your own grabbers and use them on your projects. You can find more details in How to create custom tube video grabber for KVS article. Combining grabbers and mass import Grabbers plugin can import many videos from other sites, but sometimes you may need more flexible way to grab video files and at the same time specify your own title, description, categorization and so on. In this case you can combine grabbers functionality to download files and mass import flexibility to specify data. However in this case you need to manually select videos and specify their direct page URLs, grabbing multiple videos from lists will not be possible. In order to use grabbers in mass import you should specify video page URL in Video page URL (grabbers) field. Then you can also specify any additional data you need. The key point here is that your custom fields specified in import data will override fields from grabbers. For example you can specify title and categories, so these fields will be taken from import. All other data will be taken from grabber. Using this approach will take much of your time to prepare data, but you will also benefit from more accurate and unique SEO, which is best for your project.
  3. KVS allows embedding of your content by default. In order to disable that go to Settings -> Embed player settings and remove all video formats from display slots: Your embed codes will not be showing video player anymore. Now if you also want to redirect traffic from all other sites that used your embed codes, you can add this block of code in Embed code template: <script> if (window.top != window.self) { window.top.location = '{{$data.canonical_url}}'; } else { window.location = '{{$data.canonical_url}}'; } </script> Then all users which open other sites with your embed codes will be redirected to your videos that were embedded.
  4. By default KVS comes with local storage servers for videos and albums (if ultimate package). When you add a new storage server the first thing you should ask yourself is whether you want it to be in a new server group or in an existing server group. The difference is described in details in this article: Storage system in KVS tube script Here is short summary. New server group: You want to extend storage space. You want new content to be copied to new server, while old content should remain on old server. Existing server group: You want to load balance content traffic between multiple servers. You want to migrate existing content to a new server. NOTE: content servers for videos and albums work in exactly same way, so we will only describe that for videos. How to migrate your existing content to a new server or CDN The most common scenario is when you already have local storage and want to migrate to a remote server or CDN. In order to do that you should do the following steps: Step 1. Add a new content server into the same storage group where your old server is located. Don't worry, the new server will be added in Disabled status, so it won't affect your site users. You can find more info about adding new storage servers below. Step 2. Start content sync process for the new storage server. This process can be started in servers list in KVS admin panel. It will automatically copy all content stored in the group to a new server. NOTE: sync process will create a background task and you will be able to see its progress in the list of background tasks. Based on how many content you have, this task may take days and even weeks and it will be interrupted by processing new videos if any. If you want to manually copy content files via FTP, you can do that and skip this step. Step 3. When sync process is over, if possible verify the size of content folders on both servers to make sure they have the same size. At the same time verify that KVS is not showing any errors for the new server. There will be errors until all content is synced, but there should not be any errors afterwards. Step 4. Enable new server in storage group and disable the old one. Now your users will see content from the new server. Step 5. When you are sure there are no issues, delete old server. NOTE: KVS will stop copying new videos to the old server only after you delete it from KVS. Please make sure you remember where video files are stored on filesystem before deleting old server from KVS, see step 6 for more details. Step 6. When deleting server from KVS in step 5, KVS won't physically delete any video files there. Deleting a storage server from KVS only removes it from KVS records and doesn't affect anything on filesystem. This is done for security reasons, otherwise you could accidentally remove storage server from KVS and delete all your videos files. So the only way to delete video files from the old server (e.g. free the disk space) is to do that manually from server side. We recommend you to wait for 5-7 days after deleting old storage server from KVS so that you are 100% sure that all your site is working without any issues and all videos are playing fine. When you are sure everything is OK, you are good to delete old video files from server HDD. How to add a new content server System requirements for a content server in KVS: PHP 7.1+ No specific PHP modules required Apache and / or Nginx (recommended for better performance and content protection) NOTE: Apache is NOT strictly required for remote content server, you can have Nginx + PHP. You can add a content server in admin panel in Settings -> Storage servers. For server URL you should specify any URL that you want your content to be served from. It is a common practice to use IPs and subdomains here, as well as subfolders. It is not recommended to store content in www root folder, as this may bring some problems in future. So here is the recommended URL patterns: http://123.124.125.126/videos http://storage1.domain.com/videos http://storage2.domain.com/videos NOTE: if your site works via HTTPS, all your storage servers should also use HTTPS. This is why it is better to use subdomains here, as SSL certificates can support wildcard (*) subdomains and you can install the same certificate on all storage servers. Streaming type is an important option, which indicates the type of your storage server. If this server is external CDN provider, then you should select CDN (CDN configuration for videos and albums is a topic of another article, will not be covered here). If your server doesn't have nginx installed, then you should choose HTTP 302 redirect. This option means that KVS will redirect to video files directly without any protection. Video files which are available via direct links can be easily hotlinked by other sites and someone can be stealing your traffic. This is the common issue with wordpress and other scripts, they do not allow protecting your video files. KVS keeps this option for specific cases, but its usage is not recommended. In most cases your storage server should have nginx installed and in this case you should select nginx streaming type. The next set of options are related to server connection. KVS uses these settings to copy files to remote servers. In most cases you will need to specify FTP connection details here. The most confusing option here can be FTP folder. In order to specify it correctly you should first login to your FTP using standard FTP client and see where you come into. It can be possible that your FTP is directly configured to the needed folder, but in most cases FTP will be configured into home folder of FTP user or www root folder. Since your content will be stored somewhere in subfolder (e.g. http://storage1.domain.com/videos), you should specify RELATIVE path to this subfolder in this field. For example if your FTP is configured into www root (http://storage1.domain.com), then you need to specify: videos If your FTP is configured into user's home folder, then you will need to specify more subfolders to come to /videos: public_html/videos or www/videos or even data/domain.com/public_html/videos NOTE: each storage or conversion server in KVS should have unique folder. Do not configure different servers into the same folder on filesystem, this may result in data loss. Sometimes if you specify wrong FTP folder KVS will show error that it is not able to validate connection. However in many cases there will be no error, and the files will be copied to the wrong place and will not be available to the outside. Then after you add some videos to this server KVS will show an error on it, indicating that video files are not working from this server. Control script URL field is not editable. This field is built automatically from the URL field you specified above. It indicates where you should put KVS control script on this server, so that it is integrated with KVS. Control script serves 2 things: it provides info about server (its load and disk space) and implements protection for your content. There is a link near by to download this script and you should copy it manually to the storage server so that it is available at the displayed Control script URL. So if your storage server has this URL: http://storage1.domain.com/videos Then you should copy control script here: http://storage1.domain.com/remote_control.php You can manually run this script via browser and make sure it is working and returns "connected" word. Until this happens, you won't be able to save this server. Finally, there is Time offset field. In order to implement content protection KVS generates links which expire after some time (usually after 1 or 2 hours). In order to have this working, it is required that timezones on both servers are synced. If you don't want to configure the same timezone on your remote server, then you need to specify time offset, e.g. how many hours your storage server's timezone is behind or ahead in comparison to main server. However due to different daylight saving concepts in US and Europe it can be possible that after some time this difference will be shifted 1 hour back or forth. So the best way to ensure that remote server will not have any problem is to configure the same timezone on main server and remote server. Then you will not need to specify any time offset. Here is an example configuration for a remote storage server: Finalizing protection for nginx After the server is added, open it for editing. You may see that in server settings there is a textarea with nginx config displayed (if your Streaming type is set to nginx): You should add this piece of code into nginx config of your storage server to make sure that protected files are not available via direct links. After modifying nginx config you should restart nginx. There is no common guide on where do you find your nginx config. It depends on your server panel vendor. Most server panels do allow restarting nginx from panel GUI, but do not allow modify nginx config directly. You should figure this out with your host support. But if you do not configure this, your video files will still be available via direct links and no protection will be possible. CDN storage When adding CDN storage you should use CDN option in Streaming type field. Then you also need to put your CDN control script into /admin/cdn folder (create this folder as it doesn't exist by default), specify its name in CDN control script field and specify your secret key in Streaming key field: CDN control script is a PHP script that should be implemented by your CDN provider. This script implements content protection logic by creating temporary links to video or image files. Moreover, this script is responsible for invalidating content on CDN servers when files are replaced (modified) or deleted. NOTE: invalidation process may be very important for a tube site. Consider you have DMCA claim that requires you to delete copyrighted content. If invalidation is not working correctly, your CDN will continue to stream video files that should return 404 instead. Thus you may experience troubles with providing proof of deletion. Each CDN provider has its own API for protection and invalidation, there are no standards in this field. Therefore KVS requires that each provider implements their API and provides their CDN control script to customers for use in KVS. If no such script is provided, it will still be possible to use this CDN in KVS with default CDN control script, but no content protection or invalidation will be implemented and you will be missing these important features. CDN providers: if you are willing to implement CDN control script for your CDN, just ask customer to send you default CDN control script that can be downloaded in admin panel. This script provides a basic skeleton with all the needed functions having empty or basic implementation. If you want to use default CDN control script: Download default CDN control script and upload it to /admin/cdn folder. Specify cdnapi.php in CDN control script field. Specify any text in Streaming key field as this value is not used by default control script. Troubleshooting issues KVS will validate basic issues when you try to save server settings in admin panel. After you add server, KVS will validate it on background every 5 minutes to check if it operates successfully. Even if initially you didn't have any errors, new errors may appear later due to various factors. First, FTP connection or remote server configuration may be changed externally and some functionality will simply stop working. Also when you add a server which doesn't have any videos yet, KVS is not able to check if videos are streamed correctly or not. Only after you add some videos, this check will happen on background. All issues found on background are immediately reported to KVS start page as they all are considered CRITICAL and may indicate that your remote server is not working correctly and users are not able to see videos or photos. You should make sure to fix them ASAP. Errors from background checks are not very detailed and just indicate some set of issues. In order to get more details you should try to save server settings in admin panel. NOTE: if server reports an error and you have already fixed it, please give it 5 minutes to do re-check and verify this error is fixed. Until that, server may still show this error. This only applies for background checks on existing servers. When you save server settings in admin panel, all issues are validated right a way without any delay. If saving server in admin panel still shows validation errors, these errors are not yet fixed. Here is the list of errors you can get when saving server settings: - This connection folder is already used by another server. This issue means that you are trying to configure this server into a folder, which is already used by another storage or conversion server. You should never do this, each storage or conversion server in KVS should be configured into a separate folder on filesystem. - Unable to connect to host : post. KVS is not able to connect to the specified FTP host or port. If the entered data is valid, most probably your FTP is protected by firewall and you should make sure that your main server's IP is whitelisted. Also sometimes your FTP may be configured for using non standard FTP port, you should check this as well. - Unable to login with credentials provided. Either FTP username or password is not valid. - Put / get / chmod / delete operations failed, insufficient permissions possible. When validating server connection, KVS will try to put a test file to the server and then delete it. You will get this error if any of the mentioned operations failed, which probably indicates wrong FTP folder or FTP configuration issue. Try to log in to FTP manually using desktop FTP client and try to copy any file into the specified folder. Can you do this manually? Also this error can happen if /tmp folder on your main KVS installation does not exist or is not writable. In order to check that go to Plugins -> Audit and run installation check. It will show error for tmp folder if anything is wrong. - Control script failure. As we mentioned in this article, for remote storage servers you need to manually copy control script to make it available under the displayed Control script URL. You will get this error if KVS is not able to connect to it. Try opening control script URL in browser, do you see "connected" word printed? If you don't see it, then most probably you didn't copy control script to the mentioned location, or copied it to the wrong FTP location, or there is an issue with running PHP scripts on your storage server. If script works in browser but you still see this error, it can indicate that your main server is not able to connect to this URL due to DNS or other routing issue on your main server. You can check that by using server command line and typing the following command with your control script URL: curl -I http://storage1.domain.com/remote_control.php - Remote server time is not synchronized with primary server time. As we said already, it is recommended that PHP timezones on both servers are synced. You can get rid of this error by specifying timezone offset either positive (e.g. 2) or negative (e.g. -2). This error will show timestamps on both servers to give you clue how many hours you should specify. - This script is configured with another secret key, please update its secret key. Each remote control script has secret key hardcoded in it for protection. In some cases, specifically if you use multiple KVS installations and for each of them configure the same remote storage server, you can get this error. If you already had remote_control.php script on your storage server, then most probably it is already used by other KVS installation. If you indeed have other KVS configured with the same storage server, better contact support to make sure you don't harm another installation. If you modify secret key in remote_control.php and it turns out that this file is also used by another KVS, it will stop working there. Troubleshooting content check found errors The final validation that happens for every storage server is verifying that random videos can be played. If this check fails, you will see content check found errors issue displayed for server. You can further open content test page to see test logs, which include HTTP headers logs. Here are possible reasons: Your server doesn't have nginx installed but in KVS settings you set its streaming type as nginx. In this is true, then you need to change server's streaming type option to 302 redirect. Your nginx is not configured correctly, you should ask your host support to make sure that nginx configuration printed in KVS server settings was correctly added to nginx config for this server domain. CDN API script is not working as expected or the issue is at CDN provider's end. If your server is CDN, please open content test log for this server and send HTTP headers log to your CDN provider to describe the issue. Your Apache is configured with MultiViews option enabled, which prevents KVS from serving content correctly. In this case content test HTTP headers log will show 404 error. The requested file is missing on your server and thus returns 404 error which you can see in content test HTTP headers log. In most cases this happens because your FTP configuration has been changed externally and suddenly FTP user's home folder was changed to another one. KVS started copying new files to another location and of cause these new files are no more available under the old URL, configured in KVS server settings. You should login with FTP manually and check if its location is the same as what you have specified in server URL. If the location is different, then you should check with your host support where the correct location is and ask them to move content files from this new location to the old location and re-configure FTP user back to have home folder at the old location. For example your URL is configured as http://storage1.domain.com/videos and FTP access was configured directly into this folder. As time passed, KVS put many content into this folder but then suddenly your host support changed FTP access directly to domains www folder, e.g. this one: http://storage1.domain.com. The new files are no more copied under /videos folder, but copied under / folder instead. These new files will no more be available under http://storage1.domain.com/videos URL and KVS will report content check error.
  5. Here are some important notes and prerequisites before you start In order to use HTTPS on your site you should purchase SSL certificates for your domain and all subdomains you are using with KVS project. This includes all remote storage servers as well, since any files loaded on your project, including video files should be served via HTTPS to get the full coverage. HTTPS may produce additional load on your server because of additional layer of traffic encryption. If you are using Nginx, HTTPS should be configured at Nginx level. Nginx serves static files and they need to be served via HTTPS as well. All 3rd-party scripts and advertising you are using should support HTTPS as well, otherwise they will trigger warnings in browsers and will distract your users. How to enable HTTPS in KVS tube script Step 1. Configure SSL at server level for domain and all subdomains you are using for KVS project. This should be done at Nginx level for projects that work via Nginx. If you are using CDN or remote storage servers, check how HTTPS works for all of them. NOTE: configuring HTTPS at server level is not covered by this guide. Please refer to Nginx / Apache official documentation on this. Step 2. In admin panel go to Settings -> Storage servers and for each storage server modify URL field to point to https://, e.g.: https://yourdomain.com/contents/videos Step 3. In /admin/include/setup.php change all URLs to https://. In most cases you only need to change this setting: $config['project_url']="https://www.kvs-demo.com"; However there can be other settings referring direct URLs as well. So search the whole file for http:// and change all usages to https://. NOTE: KVS will reset all cache after you change project URL. If you have much traffic on your site, your server may be overloaded for some time until the new cache is populated. Step 4. In main .htaccess file change all URLs to https:// as well. In most cases you will only have this line: RewriteRule ^(.*)$ https://www.%{HTTP_HOST}/$1 [R=301,L] Step 5. Open your site via HTTPS protocol and make sure that everything is displayed correctly. Starting from this point your site will fully work on HTTPS. Go to Plugins -> System audit and run installation and server checks there. You should not have any errors from these 2 checks. Step 6. It is recommended to configure 301 redirect for all HTTP requests to HTTPS so that there is no duplicate content available under different protocols. This redirect should be set globally in Nginx configuration (or Apache configuration if your server is not running under Nginx). NOTE: configuring redirect at server level is not covered by this guide. Troubleshooting Design is not displayed properly when using HTTPS. Most probably you didn't switch all static resources to be loaded via https://. Open source code of your page and search for http:// text. Once you located the URL of static resource referred by http://, go to Website UI -> Search in templates in admin panel and try searching for template which renders this URL. Fix to https://. On some pages browser shows insecure warning. This problem is caused by the same issue as above. Some parts of your page are trying to use resources via insecure HTTP protocol and that's why browser shows the warning.
  6. Starting with KVS Advanced you can show video previews for every video in the list instead of single or scrolling thumbs. Here are some important notes and prerequisites before you start If you have many videos, creating previews can take much time, up to many months! You should make sure that each storage server has enough space to store up to 1mb for each video. Each preview will take 600-800kb, so showing this for every video will increase your traffic usage. Steps on how to enable video previews Step 1. In admin panel go to Settings -> Add video format and create new video format with the following data: Title: MP4 preview Postfix: _preview.mp4 Video type: Standard videos Status: Active optional (you can set Active required here if you don't have videos or have only few videos) Video size: Resize video according to: NxM Enfore this size (replace NxM with exact size you need video to be displayed in lists) FFmpeg options (-crf 28 is a compression quality, use lower number for higher quality): -vcodec libx264 -movflags +faststart -threads 0 -r 25 -g 50 -crf 28 -me_method hex -trellis 0 -bf 8 -an -f mp4 Limit duration to: 20 seconds (or any number you want it to be) Number of parts: 4 (or any number of parts you want it to be) Other fields under Duration limitation section - as per your needs. Save it. NOTE: if you selected Active required status and you have any videos, KVS will automatically create tasks for creating these files for all your videos. You should use Active optional status if you have many vids. You can see this video format in action here in KVS tube script demo admin panel: http://kvs-demo.com/admin/formats_videos.php Step 2. Now it is time to create video previews for some test videos. Please skip this step if you already selected Active required status for preview video format. Go to Videos list. Select some test videos (the ones you can see on your latest updates page) and under Batch actions select Mass edit selected. You will see the mass edit GUI. Scroll down to Content section and select MP4 preview format under Create / Re-create format files: NOTE: Make sure you didn't select anything else in the big set of options. Apply changes. In Administration -> Background Tasks section you can now see list of tasks to create preview files for the selected videos. You should wait until they are processed, this may take a while. If there are errors on these tasks, please contact support. Step 3. Go to Website UI -> Theme Settings and select MP4 Preview format in Video preview format option. Step 4. If preview creation tasks are finished at this point, go to your latest videos and try mouse over the test videos you've chosen to create preview files. You should see the video previews playing. If you don't see them, try refreshing your browsers cache with Ctrl+F5. Step 5. If you are OK with the quality and functionality of your test previews, you can enable them to be created for all videos now. Go to Settings -> Video formats -> MP4 Preview and change status to Active required. NOTE: this will create tasks for all your videos, which can take long time. All new videos will have this format automatically. Styling waiting icon You can modify the following CSS style in your site's style file: .list-videos .img.preview-loading:after { display: block; position: absolute; content: ''; width: 25px; height: 25px; background: #000000 url('../images/loader.gif') 2px 2px no-repeat; bottom: 5px; right: 5px; z-index: 1; } Troubleshooting It is not possible to create a new video format. This is not possible if you have KVS basic license. Basic licenses do not support multiple formats and will not allow you using this feature. Tasks for creating preview files failed. Please create ticket for support, there can be many reasons and need investigation in your admin panel. Video previews are not displayed. Are you trying on the test videos you have selected for massedit? Please open them in admin panel and check if they have preview formats created: Did you try resetting browser cache with Ctrl+F5? Are you using the same browser where you logged in admin panel? If using different browsers to test this functionality you should log in to admin panel in each browser. Otherwise you can see the cached page version which has not yet been updated with this new feature.
×
×
  • Create New...