Jump to content

Search the Community

Showing results for tags 'grabbers'.

  • Search By Tags

    Type tags separated by commas.
  • Search By Author

Content Type


Forums

  • General Discussion
    • Announcements
    • Business Discussion
    • Business Services
    • Sell and Buy
  • Themes & templates
    • KVS default theme
    • KVS tablet theme
    • KVS paysite theme
    • KVS basic theme 1
    • KVS basic theme 2
    • KVS basic theme 3
  • Educational / support
    • Technical Support
    • FAQ
    • Educational Series

Find results in...

Find results that contain...


Date Created

  • Start

    End


Last Updated

  • Start

    End


Filter by number of...

Joined

  • Start

    End


Group


About Me

Found 7 results

  1. We are Adult-HTML, EU based company and a friendly team of 20+ experienced web developers and designers, that builds fully-functioning websites and implements custom solutions since 2009. We provide whole package of services you need and have perfect experience with: UI/UX design (Tubes, Paysites, Affiliate programs, Blogs, Webcam sites etc.) HTML5 / CSS3 / JS (Responsive, tidy, cross-browser compatible). Integration (KVS, AVS, Mechbunny, WP or other scripts) Customized scripts (Laravel and other PHP frameworks) QA team (Human manual testing: popular OS and real devices) Responsive Project Management team. We reply queries as quick as we can during our working hours: from Monday to Friday 7am-4pm UTC. Flexible collaboration. We are ready to join your team and get in your workflow processes. Dedicated team or single web developer/designer can become a part of your team, so you are able to manage them directly (on a monthly basis rate). We also offer on-demand partnerships (on hourly basis rates) and able to develop full project/product from scratch, fix issues after previous negligent developer, impact on site optimisation (based on your or our report) and provide further lifetime support for your project. Some Key Facts about us: 12+ years of experience, 420+ completed KVS sites, 2900+ delivered projects, 100% job success score, 150+ excellent reviews on GFY and Master-X. Please check our portfolio and don’t hesitate to contact us via contact form at https://adult-html.com Email: adulthtml@gmail.com Skype: adult-html Telegram: @adulthtml
  2. Hi, As the topic states, I am looking for someone to help me create a couple of grabbers for tube sites that do not have pagination, but rather scroll down to load more options. Can someone help build these kind of grabbers? thanks.
  3. Do you have adult grabbers already made for other sites? I am buying grabbers!
  4. Is there any way to drop a YouTube video url into the admin, then have it parse the video info automatically? Looking for a way to drop like ten links in at a time (as fast as possible), have it grab the titles/description and embed the video, and then just have to edit them accordingly. Does anyone have a front end "add embedded video" option on their site for webmaster users?
  5. KVS provides API to use youtube-dl server library for scrapping videos from other tube sites. You can implement your own grabber class in PHP language and upload it into KVS. Here is how this can be done. The example features fully working custom youtube grabber (KVS has built-in grabber for youtube by the way). NOTE: it is not strictly required to use youtube-dl API, it is also possible to create a completely custom grabber with your own code. Implementing grabber class using youtube-dl API Create CustomGrabberYoutube.php with the following code (also attached here as a text file): <?php // when you change classname, change it at the very bottom as well in this line: // $grabber = new CustomGrabberYoutube(); class CustomGrabberYoutube extends KvsGrabberVideoYDL { // =============================================================================================================== // infrastructure methods // =============================================================================================================== public function get_grabber_id() { //prefix your grabber ID with "custom_" return "custom_videos_youtube"; } public function get_grabber_name() { // name displayed in admin panel return "youtube.com"; } public function get_grabber_version() { // this is required for grabbers that are autoupdated from KVS return "1"; } public function get_grabber_domain() { // domain name, KVS will check this to find out if this grabber is suitable for the given URL return "youtube.com"; } public function get_supported_url_patterns() { // returns list of regexp patterns that describe video URLs, for youtube this pattern will match // https://www.youtube.com/watch?v=htOroIbxiFY return array("/https?:\/\/(www\.)?youtube\.com\/watch.*/i"); } public function can_grab_description() { // return true if your grabber is going to provide description for each video return false; } public function can_grab_categories() { // return true if your grabber is going to provide categories for each video return false; } public function can_grab_tags() { // return true if your grabber is going to provide tags for each video return false; } public function can_grab_models() { // return true if your grabber is going to provide models for each video return false; } public function can_grab_content_source() { // return true if your grabber is going to provide content source for each video return false; } public function can_grab_date() { // return true if your grabber is going to provide date for each video return false; } public function can_grab_rating() { // return true if your grabber is going to provide rating for each video return false; } public function can_grab_views() { // return true if your grabber is going to provide views for each video return false; } public function can_grab_video_files() { // this should be true for youtube-dl return true; } public function get_supported_qualities() { // list of supported video qualities, should match what youtube-dl returns in its info under formats // run this command: // youtube-dl --dump-json https://www.youtube.com/watch?v=PhDXRCLsqz4 >> test.json // and open test.json in Firefox, find "formats" array and look into the available formats // youtube has too many formats, KVS only supports formats with "ext"="mp4" // you can list them here and you will be able to select from them in grabber settings return array('360p', '720p'); } public function get_downloadable_video_format() { // for youtube-dl grabber KVS only supports mp4 formats return 'mp4'; } public function can_grab_lists() { // return true if you want to allow this grabber to grab lists and thus be used on autopilot // if true, you will also need to implement grab_list() method - see below return false; } // =============================================================================================================== // parsing methods - modify if you need to parse lists or add additional info // =============================================================================================================== public function grab_list($list_url, $limit) { // this method is used to grab lists of videos from the given list URL // $limit parameter means the number of videos to grab (including pagination) // if $limit == 0, then you just need to find all videos on the given URL, no need to care about pagination $result = new KvsGrabberListResult(); // $page_content here is the HTML code of the given page $page_content = $this->load_page($list_url); // parse $page_content and add all video URLs to the result // consider pagination if needed // you can use $this->load_page($list_url) method to get HTML from any URL $result->add_content_page("https://youtube.com/video1"); $result->add_content_page("https://youtube.com/video2"); $result->add_content_page("https://youtube.com/video3"); return $result; } protected function grab_video_data_impl($page_url, $tmp_dir) { // by default the base class will populate these fields (if provided by youtube-dl): // - title // - MP4 video files for the qualities listed in get_supported_qualities() function // - description (should be enabled in can_grab_description() function) // - date (should be enabled in can_grab_date() function) // - tags (should be enabled in can_grab_tags() function) // - categories (should be enabled in can_grab_categories() function) $result = parent::grab_video_data_impl($page_url, $tmp_dir); if ($result->get_error_code() > 0) { return $result; } // do any custom grabbing here for additional fields, which are not supported by youtube-dl // $page_content here is the HTML code of the given video page //$page_content = $this->load_page($page_url); // parse HTML code and set additional data into $result, e.g. data which is not provided by youtube-dl //$result->set_rating(85); //$result->set_votes(10); //$result->set_views(123874); //$result->set_content_source("Content Source Name"); //$result->add_model("Model 1"); //$result->add_model("Model 2"); return $result; } } $grabber = new CustomGrabberYoutube(); KvsGrabberFactory::register_grabber_class(get_class($grabber)); return $grabber; The code has comments where needed. Basically youtube-dl provides main video info, such as title, description, tags, categories, date and files. If this is enough for you, you should only modify set of methods on top grouped under infrastructure methods section. These methods are designed to integrate grabber into KVS, so you should change them as described. You should also modify grabber class name in 2 places (top and bottom) and make sure that grabber class name is unique and has Custom in its name (to avoid intersections with any future grabbers we will add). If you want to implement parsing lists or add additional info, you should modify parsing methods as explained in the code. Implementing grabber class without youtube-dl Here is example grabber class that is not using youtube-dl. Put your custom parsing logic: <?php // when you change classname, change it at the very bottom as well in this line: // $grabber = new CustomGrabberYoutube(); class CustomGrabberYoutube extends KvsGrabberVideo { // =============================================================================================================== // infrastructure methods // =============================================================================================================== public function get_grabber_id() { //prefix your grabber ID with "custom_" return "custom_videos_youtube"; } public function get_grabber_name() { // name displayed in admin panel return "youtube.com"; } public function get_grabber_version() { // this is required for grabbers that are autoupdated from KVS return "1"; } public function get_grabber_domain() { // domain name, KVS will check this to find out if this grabber is suitable for the given URL return "youtube.com"; } public function get_supported_url_patterns() { // returns list of regexp patterns that describe video URLs, for youtube this pattern will match // https://www.youtube.com/watch?v=htOroIbxiFY return array("/https?:\/\/(www\.)?youtube\.com\/watch.*/i"); } public function can_grab_description() { // return true if your grabber is going to provide description for each video return true; } public function can_grab_categories() { // return true if your grabber is going to provide categories for each video return true; } public function can_grab_tags() { // return true if your grabber is going to provide tags for each video return true; } public function can_grab_models() { // return true if your grabber is going to provide models for each video return true; } public function can_grab_content_source() { // return true if your grabber is going to provide content source for each video return true; } public function can_grab_date() { // return true if your grabber is going to provide date for each video return true; } public function can_grab_rating() { // return true if your grabber is going to provide rating for each video return true; } public function can_grab_views() { // return true if your grabber is going to provide views for each video return true; } public function can_grab_video_files() { // return true if your grabber is going to provide video files for each video return true; } public function can_grab_video_embed() { // return true if your grabber is going to provide embed code for each video return true; } public function can_grab_video_duration() { // return true if your grabber is going to provide duration for each video return true; } public function can_grab_video_screenshot() { // return true if your grabber is going to provide screenshot for each video return true; } public function get_supported_qualities() { // list of supported video qualities that your grabber provides return array('360p', '720p'); } public function get_downloadable_video_format() { // only grabbers that return MP4 files are supported return 'mp4'; } public function can_grab_lists() { // return true if you want to allow this grabber to grab lists and thus be used on autopilot // if true, you will also need to implement grab_list() method - see below return false; } // =============================================================================================================== // parsing methods // =============================================================================================================== public function grab_list($list_url, $limit) { // this method is used to grab lists of videos from the given list URL // $limit parameter means the number of videos to grab (including pagination) // if $limit == 0, then you just need to find all videos on the given URL, no need to care about pagination $result = new KvsGrabberListResult(); // $page_content here is the HTML code of the given page $page_content = $this->load_page($list_url); // parse $page_content and add all video URLs to the result // consider pagination if needed // you can use $this->load_page($list_url) method to get HTML from any URL $result->add_content_page("https://youtube.com/video1"); $result->add_content_page("https://youtube.com/video2"); $result->add_content_page("https://youtube.com/video3"); return $result; } protected function grab_video_data_impl($page_url, $tmp_dir) { $result = new KvsGrabberVideoInfo(); // $page_code here is the HTML code of the given video page $page_code = $this->load_page($page_url); if (!$page_code) { $result->log_error(KvsGrabberVideoInfo::ERROR_CODE_PAGE_UNAVAILABLE, "Page can't be loaded: $page_url"); return $result; } // parse HTML code and set data into $result // replace with your parsing logic $result->set_canonical($page_url); $result->set_title("Demo title"); $result->set_description("Demo description long description long description long description long description."); $result->set_screenshot("http://www.localhost.com/test/test.jpg"); $result->set_duration(30); $result->set_date(time()); $result->set_views(1526); $result->set_rating(87); $result->set_votes(11); $result->set_embed("<div>embed code</div>"); $result->add_category("Category 1"); $result->add_category("Category 2"); $result->add_category("Category 3"); $result->add_tag("Tag 1"); $result->add_tag("Tag 2"); $result->add_tag("Tag 3"); $result->add_model("Model 1"); $result->add_model("Model 2"); $result->add_model("Model 3"); $result->set_content_source("Content Source 1"); $result->add_video_file("360p", "http://www.localhost.com/test/test_360p.mp4"); $result->add_video_file("720p", "http://www.localhost.com/test/test_720p.mp4"); $result->add_custom_field(1, "Custom1"); $result->add_custom_field(3, "Custom3"); return $result; } } $grabber = new CustomGrabberYoutube(); KvsGrabberFactory::register_grabber_class(get_class($grabber)); return $grabber; Testing grabber class Put grabber class file to your project root folder. Also create test_grabber.php file in the same folder with the following code: <?php header('Content-Type: text/plain; charset=utf8'); ini_set('display_errors', 1); error_reporting(E_ERROR | E_PARSE | E_COMPILE_ERROR); require_once('admin/plugins/grabbers/classes/KvsGrabber.php'); $grabber = require_once('CustomGrabberYoutube.php'); $grabber->init(new KvsGrabberSettings(), ""); if ($grabber instanceof KvsGrabberVideoYDL) { $grabber->set_ydl_binary('/usr/local/bin/youtube-dl'); } print_r($grabber->grab_video_data('https://www.youtube.com/watch?v=htOroIbxiFY', 'tmp')); Modify this code to your class name and specify your demo URL. Then run via browser: http://domain.com/test_grabber.php If everything is fine, you should see dumped info from the scrapped video. Installing grabber into KVS Just go to Plugins -> Grabbers in admin panel and upload your grabber class into Custom grabber field. Then after saving the form you will see your grabber installed marked with red color. You need to open this grabber settings and select Content mode = Download. Also enable the needed fields under Data. NOTE: If you don't see any fields under Data, then your grabber class doesn't return true from can_grab_xxx() methods. If you want to update grabber class, simply upload it again. It is recommended to increment version in get_grabber_version() method to stay sure on which version KVS is using. Finding the list of supported video files to grab If you don't know which formats source site provides (usually a subset of: 240p, 360p, 480p, 720p, 1080p), you can check that from youtube-dl: youtube-dl --dump-json https://www.youtube.com/watch?v=PhDXRCLsqz4 >> test.json This should generate test.json file which can be open in firefox to show JSON structure. Find a node called formats, it should be a list with items describing each supported format. KVS can only import formats with ext = mp4, you can list them in get_supported_qualities() method using XXXp notation, e.g. 360p, 720p. Here is sample screenshot for youtube: CustomGrabberYoutube.txt
  6. KVS admin panel supports all possible ways to add videos. All you need is to stick to the way which best suits your situation, whether you have existing content or not. There are different content types supported for videos (upload, embedding, hotlinking and other), you may also need to understand how they are different from each other in What video types are supported in KVS tube script and how they are different post. When using upload content type, it is also possible to upload files differently depending on whether you want them to be re-converted or not, please check Different ways to upload video files into KVS for more details. Adding videos manually This is the most obvious way to add videos. The form can be found in Videos -> Add video section of admin panel. You can specify all possible options there and send new video to conversion engine. You don't need to log in to website area as a specific user to add videos from this user - you can do that in admin panel by selecting the needed user in add video form. In Settings -> Content Settings section under Video adding / editing settings you can select some options that affect this form: the most valuable are default user and default status. Based on which video content type you want to add, KVS will show only fields that are related to the selected content type: Adding videos manually is very simple and straightforward process, but this won't work well if you need to add many videos. Therefore please check different mass importing options available in KVS. Mass importing videos from a text file or Excel spreadsheet This feature should be used in the following cases: You have a text file created by some other website or program with the needed list of videos to import (KVS export feature can also generate such text file to export videos from one KVS to another). You have a list of video files uploaded to your server and you want to import them into KVS with specifying all additional info. For accessing this functionality you should go to Videos -> Import videos. There you can copy-paste your text field into a data textarea, make sure to configure Field separator correctly (\t for copy-paste from Excel, semicolon (;) for CSV, or other separator if you are using plain text file). In most cases Line separator should not be changed. Then you need to select list of Fields in the exact order you have in your import data. IMPORTANT! Some CSV files may have separator at the end of every line, which actually means that there is one more empty field at the end of every line. In this case you should add one more additional field and select * Skip this field * option there. If you still have error indicating wrong number of fields in every single line, you should carefully check whether you've selected the correct separator and whether the number of fields matches the number of separators in the line + 1. The number of fields should be +1 all the time! Here are some examples. Single text file import with pipe (|) separator: Specifying data in Excel and then copy-paste: When using import functionality it is very important to understand which data you need to provide based on which video content type you are trying to import. Video upload: you should either provide video file URL in Source video file (with processing) field, or one / several Video file "XXX" (without processing) fields. Video hotlink: you should provide video file URL in Hosted file URL (hotlinking) field. Additionally you may specify Duration and Overview screenshots (sources comma-separated). If you don't provide duration or screenshots in import data, KVS will have to download the file you are hotlinking for detecting its duration and screenshots. This may significantly increase video processing time. Video embed: you should specify Embed code (embedded video) field. In addition if you also specify Hosted file URL (hotlinking) field, KVS will temporarily download file from this URL and will detect duration and screenshots automatically from it. Otherwise if you don't know video file URL or you don't want KVS to spend time on its download, you should specify Duration and Overview screenshots (sources comma-separated) fields. Pseudo video: you should specify Outgoing URL (pseudo video) field. In addition if you also specify Hosted file URL (hotlinking) field, KVS will temporarily download file from this URL and will detect duration and screenshots automatically from it. Otherwise if you don't know video file URL or you don't want KVS to spend time on its download, you should specify Duration and Overview screenshots (sources comma-separated) fields. Import feature supports the huge list of other options, but for novice users in most cases you don't need to touch them and better use everything by default. If you import many videos, you may need to look at post date randomization options, so that you can import videos to be published in some future or past period. For example you can import 1000 videos as they have been added one by one since 2010. Anyway, you can always do the same via massedit feature, so not something should be really important during import phase. After you submit the import, KVS will validate each single line and will show you summary so that you can confirm it. The validation may take much time depending on your input data. By default KVS validates all URLs to be valid, which may take some time. You can disable URL validation in import options. It is recommended to start with several lines first to make sure you understand the concept. When looking into import validation summary you will be able to see all errors and warnings. Warnings are not critical, they just indicate potential issues in your input data, which are not necessary real issues. Errors are critical and indicate the lines that can't be imported. If there are any lines without errors, you can still confirm import and lines which have errors will be simply skipped from being imported. If all lines have errors, you will not be able to confirm the import. After you confirm import summary, KVS will create a background task for import. You can see import tasks in Administration -> Imports log. There you can see data submitted with each import as well as import status and log. All imports are processed on background. You don't have to wait for anything here. Import tasks may have errors, even if validation didn't find any issues. There may be issues with downloading files, network issues and some other. IMPORTANT! If you submitted an import but you don't see any videos added, you may need to go to Administration -> Imports log and look at what is going on there. It may be that all import lines failed. Looking into import log will help you understand the issue. It may also be a case that your import is stuck due to lack of free space on your main server (since import in most cases download new files to your server there is a protection to check disk space, so that your HDD is not 100% filled). In this case import will automatically continue as soon as more free space is available. When you use this functionality really often, it may help to create an import pattern, so that you do not select the same options every time. You may define as much different import patterns as needed, then you just need to select the needed pattern and copy-paste import data. The rest of options will be defaulted from import pattern. Uploading video files via FTP and use FTP content uploader plugin If you already have a list of video files and want to quickly upload them into KVS, the easiest way is to use FTP content uploader plugin. You just need to create a folder somewhere under your www root and upload all your video files there. Then in FTP content uploader plugin you should specify full server path to that folder, or relative path under your www root (not starting with /, e.g. upload for http://domain.com/upload folder). Each video file in this folder will be treated as a standalone video and will be uploaded into KVS with just a few clicks: In advanced scenario here if you want to upload multiple formats for the same video as well as screenshots and / or description, you may need to create subfolders for each video. All files in a subfolder will be considered to belong to each video. The following rules will be applied: Subfolder name means title of a video. ZIP file means list of screenshots uploaded for a video. JPG file means main screenshot uploaded for a video. TXT file means description specified for a video. Video files in subfolder may be treated differently based on your selection in Video formats option: -- If you want to upload as a source file with processing, then the biggest video file will be chosen as a source file, other video files will be ignored. -- If you want to upload as format files without processing, the plugin will compare file names with video formats to find postfix matching. As a result, plugin will show you a summary of all detected files and after you confirm that, they will be uploaded as videos into KVS. It is recommended to enable delete option (if not a problem that files will be deleted), because this will use rename operation on filesystem, which works instantly in comparison to copy operation. Using this plugin will not allow you to specify other data, such as categories, tags and etc. It is expected that you want to create videos first and then you may manually edit them in admin panel, after they are imported. If you want to specify all this data at import time, you need to consider using import feature (2). Since your video files are already uploaded on your server, you just need to specify their URLs for the import to take them. At the same time you can also specify other data needed. Automating mass importing videos with a feed This feature should be used in the following cases: Import data is regularly updated and you want to automate importing the updates. You want to automatically sync content between 2 or more KVS projects. You have a software that uploads video files to your FTP and you want then to auto-import them (for example CCTV or webcams). To access this functionality go to Videos -> Add importing feed section. Feeds are very similar to mass import described in previous section, but the main purpose of a feed is to automate importing without your intervention. Every feed can be configured to query a text file from the given URL every given interval and add all new content that appears in this text file. Many options here are very similar to mass importing options: for plain text file format you need to specify field and line separators and select list of data columns. When a feed gets executed for the first time, it will add all available content from the given file if you don't specify any limit. Starting from the next execution feed will only add new content that appears in the text file. Here it is very important to understand how duplicates are being calculated. First of all you should specify Duplicates prefix - any unique string for this feed. Then if using CSV / plain text format you should also indicate the key field from this file - a field which is unique for every line. It can be video ID, or video URL or title - any unique field. Duplicates prefix is needed for cases when you need to separate content between different feeds. Imagine that you used video ID as a key field. If you import data from multiple providers by using multiple feeds, IDs may overlap there, e.g. video #1234 may exist at both provider 1 and provider 2. If KVS only used ID to identify duplicates, it would end up by considering video #1234 from provider 1 and video #1234 from provider 2 as duplicates, so one of these videos wouldn't get added. But if you specify Duplicates prefix unique for each feed - this will not be a problem anymore. Video #1234 from feed 1 will not be a duplicate of video #1234 from feed 2. At the same time, you may have several different feeds adding content from the same source (for example 1 feed is pulling all videos from Category 1 and another feed is pulling all videos from Category 2). Since these feeds pull content from the same source and the content may overlap, you should specify the same value for Duplicates prefix for both feeds. Because a single video #1234 may have both Category 1 and Category 2 and thus it will appear in both feeds. If you've specified different prefixes, this video will be added twice. But if prefixes are the same - videos #1234 in both feeds will be considered as duplicates. Grabbing videos from 3rd party sites KVS features extremely powerful tube grabbers plugin. With this plugin you can import tons of content, even if you don't have any. We will not speculate on whether it is ethic or not to grab content from 3rd party sites, you should decide this yourself. In order to use grabbers you first need to install and configure some. KVS provides a list of most popular tube grabbers and from time to time we expand it with new grabbers. There is also a default video grabber which can download videos from abstract tube sites. However specific grabbers are more powerful: They may support embed codes if source site provides them. They may support additional video data, such as categories, tags, models, rating, views, date, etc. They may support downloading video files in multiple qualities, so that you don't need to convert them on your server. Most of video grabbers require youtube-dl library to be installed on your server. If you don't have it, please ask host support to install it for you, otherwise it won't be possible to use many grabbers. After you install some grabbers, you should configure them. You should select Mode option to indicate whether you want this grabber to download, embed or add videos as pseudo videos. Then you also need to choose which data fields you want to grab; the set of fields is different for each grabber and is based on which data can be parsed from the grabbed site. Finally there are filtering options, that help you to maintain grabbed content quality. IMPORTANT! when you want to use grabbers to download videos to your server, like with any other methods you can decide whether you want the downloaded video files to be processed, or you want them to be saved without processing. Here you can use 2 different tactics based on whether your project has enough computing power or not: If your project doesn't have much computing power, it makes sense to download all files without processing. In order to configure this you should set Multiple value for Quality option and then based on the number of video files provided by grabber match your video formats: If your project has enough computing power, it may be faster for you to download only 1 single file from grabber and have other files created by KVS. This may work faster, because download from grabbers is typically slow due to speed throttling by the source site. In this case you should select the needed file under Quality option and assign it to the correct format; so that all other formats will be created by KVS: NOTE: in some cases if you want to process all files, even the downloaded one, you will need to set upload as Source file. One important issue you'll find when using grabbers is how to manage your categorization efficiently. Different sites may have many similar categories named differently, so that grabbers may create too many categories in your database. There are several workarounds to deal with this described in Categorization best practices article. There are 2 ways you can import videos from grabbers plugin: you can import videos manually or you can configure grabbers to ping video lists from time to time and add videos from these lists. When using grabbers for the first time we recommend running manual imports first to understand how grabbers work and whether the grabbed videos are in the way you want them to be added. In order to import videos manually you should go to Upload content using grabbers manually section in grabbers plugin. There you need to specify list of URLs you want to grab. In this list you can specify both individual videos (if you want to pick each video yourself) and also you can specify video list URLs or video RSS URLs to grab the whole lists. Here is where you can feel the real power of using site-specific grabbers - you can use your browser to navigate source site and find a page which lists the needed videos - this can be a page listing all top rated videos, or most viewed videos for the last week, or videos from a specific category, or videos from a specific user and etc. You can then copy-paste URL you see in browser address field and grabber will open that page and will add all videos from it. Many tube grabbers also support pagination mode, which means that grabber will be able not only to add all videos you see on that URL, but also to navigate to page #2, to page #3 and etc. of a paginated list and find all videos there as well. In order to enable that, you need to specify a number of videos you want to grab right after the URL separated by pipe symbol, e.g.: https://www.kvs-demo.com/top-rated/|1000 This command tells KVS grabber to import 1000 videos from this page. If this page is a paginated list and shows only 50 videos per page, and the grabber supports pagination, KVS will query 20 more pages from this list to add 1000 videos in total. Using this approach you can mass import 100 top rated videos from every category on the site by simply specifying URLs to category pages and number of videos you want to grab from them. If you do not specify a number after list URL, KVS will add only those videos that are displayed initially on this page, e.g. only 50 videos for this sample top videos list: https://www.kvs-demo.com/top-rated/ Finally, when you have imported sample list of videos and you are happy with them, you can move forward and configure auto-pilot. Auto-pilot is exactly the same as importing manually via specifying list URLs and counts. The only difference is that auto-pilot will do that automatically for you, e.g. it will query the specified pages every N hours as you configure and will check if there are any new videos there. Therefore if you want it to import new videos every time, you should make sure to provide list URLs, which are sorted by new videos first. If you recall our previous example the top.html will not work here, since top lists are rarely updated. Instead in auto-pilot settings you should specify URLs which are updated every N hours you plan to query for new videos, for example latest videos page: https://www.kvs-demo.com/latest-updates/|100 By setting the above URL to auto-pilot you will require KVS to grab new videos from this website every N hours. Then you simply do not need to upload any videos manually, because they will be added and added every N hours you configure (if the source site is updated of cause with the new videos). If you want to grab only from specific categories, or tags, or models or whatever, you will need to specify multiple URLs, each for the list you want to grab, e.g. like this for grabbing from 3 categories: https://www.kvs-demo.com/categories/rock-music/|100 https://www.kvs-demo.com/categories/house-music/|100 https://www.kvs-demo.com/categories/pop-music/|100 While you can also use RSS with grabbers, using the URLs gives you much more power than RSS. In most cases you can adjust sorting or even do some filtering in the URL, so that grabbers only import content you'd like to appear on your site. Here are some important highlights with using tube grabbers in KVS: Use specific grabbers designed for specific sites, they support more options. If no specific grabber is available for the site you want to parse, you can try using default video grabber if it suits you (this grabber only supports download mode). But this grabber won't support 100% sites, it can only grab from sites that do have video URLs clearly printed in HTML code. When downloading from other sites, your IP may be blacklisted for frequent downloads. KVS has no control on that other than letting you specify timeout to avoid frequent requests. Some grabbers support proxies that can be listed in their settings. Try importing manually first before you configure auto-pilot. If you want to grab initial set of content, grab it manually and use auto-pilot for pulling new videos. Use web URLs instead of RSS to download from video lists. Make sure videos in the list are filtered and sorted in the way you want to grab them. You can still use RSS if you want to. Default grabber only supports RSS lists. Grabbers may become broken from time to time if source sites gets updated. If you find this issue, please report to KVS support and we will update the broken grabber to be working again. With KVS you can create your own grabbers and use them on your projects. You can find more details in How to create custom tube video grabber for KVS article. Combining grabbers and mass import Grabbers plugin can import many videos from other sites, but sometimes you may need more flexible way to grab video files and at the same time specify your own title, description, categorization and so on. In this case you can combine grabbers functionality to download files and mass import flexibility to specify data. However in this case you need to manually select videos and specify their direct page URLs, grabbing multiple videos from lists will not be possible. In order to use grabbers in mass import you should specify video page URL in Video page URL (grabbers) field. Then you can also specify any additional data you need. The key point here is that your custom fields specified in import data will override fields from grabbers. For example you can specify title and categories, so these fields will be taken from import. All other data will be taken from grabber. Using this approach will take much of your time to prepare data, but you will also benefit from more accurate and unique SEO, which is best for your project.
  7. KVS supports different content types for videos (uploading, embedding, hotlinking and other), here is full information about them: What video types are supported in KVS tube script and how they are different. When using upload content type, you upload a source video file into KVS and then let KVS process it and store it using KVS storage system. During video processing phase KVS will create all required video formats defined in your settings. KVS will also try to create all conditionally-optional video formats, if the uploaded source file meets their size and duration limits. The most common example here is when you have multiple video formats defined for multiple qualities, e.g. 360p, 480p and 720p. In such scenario 360p should be configured as required format, 480p and 720p should be configured as conditionally-optional formats. So that 480p and 720p are only created for source video files that allow resizing them to 480p or 720p respectfully. For example if you upload source file in 720p - all 3 formats will be created. If you upload source file in 480p - only 360p and 480p will be created, 720p format will be skipped. For source files with 360p or less quality KVS will only create 360p format, because this is the only required format in this configuration. When using the described approach you don't need to think about source file original quality, all you need is to upload it as a source file with processing and KVS will then create all necessary formats. This processing will however take much time, because operation to create one video file from another uses your server resources very intensively. There can be scenarios when you will want to skip this processing: You don't need to have multiple video formats, you need to show video in only one quality. You already have video files in the needed format(s), for example you already prepared them using desktop software or you import existing video files from external sources. You don't need any resize or watermarking on video files. KVS provides ability to skip video processing for all or some files. In order to do that you should upload your video files not as source files, but directly into the needed video format(s). When a file is uploaded as a video format, KVS will not do any re-conversion with this file, the file will be saved in the same form as you've uploaded it. By doing this you can significantly decrease video processing time, but there are certain limitations you should be aware of: No watermarking will be applied to the uploaded file as well as no resize and duration trim. You should make sure that the uploaded file has valid format (e.g. if you upload it into MP4 format, the file should be a valid MP4 file, not any other video file). If you try to upload an invalid file, most probably there will be an error thrown during processing phase, however this depends on your ffmpeg version and it is also possible that there will be no error and you will end up with a video that is not playable via web player even if you can play it with your desktop. If you have multiple video formats, you can choose to upload them all and fully skip conversion, or upload only one of them and let KVS create the rest. Depending on which functionality you typically use to upload videos into KVS, they all provide ability to upload in both ways. Naming convention for this option is as following: Source file (with processing): means that the uploaded video file will be considered as a source for all video formats that will be created by KVS. Format "XXX" (without processing): means that the uploaded video file will be uploaded directly as the given format, so it won't be processed by KVS (other formats may be created from it if required). Manual video adding When adding videos manually you can see a field to upload as a source file (Source file) and separate fields to upload into each specific video format (e.g. Format "XXX"): Videos mass import In mass import for each field you can either select if it should be considered as a source file (Source video file (with processing)), or as a specific video format (Video file "XXX" (without processing)): Videos importing feeds In importing feed settings you can choose whether all video files from this feed should be considered as source files (Source file (with processing)), or as files of a specific format (Format "XXX" (without processing)). Importing feeds do not support uploading multiple files in multiple video formats: FTP content uploader plugin Using FTP content uploader plugin is quite tricky if you have multiple video formats and want to upload them as formats without processing. In order to do that you should name your video files based on postfixes you have in video formats. For example if you want to upload only 1 format per each video, you can still put all video files into 1 upload folder, but make sure that their file extension match the postfix of the video format you want it to be uploaded to, e.g. test.mp4 will match video format with .mp4 postfix. Also you should set Video formats option to detect postfixes: If you want to upload multiple video formats for each video, then you should put files for the same video into a subfolder under your upload folder. Each file should have filename based on video format postfix you want it to be uploaded to, e.g. test.mp4, test_480p.mp4 and test_720p.mp4 for formats with .mp4, _480p.mp4 and _720p.mp4 postfixes. Then on summary page you can check if all the formats have been detected as expected: Grabbers plugin In grabbers plugin if you want to download only 1 video file, you can simply select whether it should be considered as a source file or specific format file under upload as option: If the source site provides multiple video formats, you can download as many of them as needed and upload as specific video formats. In order to get these fields displayed you should switch Quality field to Multiple value: NOTE: downloading multiple files from grabber could be slowly due to bandwidth throttling on the source site. If your project's conversion power is high, it may be faster to download only 1 file and then create other formats via KVS. Summary Uploading as a source file with processing: Should be used by inexperienced users. You can upload any source file with any quality (480p, 720p and etc), KVS will automatically decide which formats to create. KVS will create all the needed formats with all their settings, e.g. watermarks, quality, duration trim. Works slow due to video conversion utilized. Uploading as a format file without processing: Should be used by experienced users if all restrictions are not a problem (you don't need watermark and etc. see below). Video file format and quality (480p, 720p and etc.) should exact match video format settings. No watermarking, quality change and duration trim will be done. Works incredibly faster.
×
×
  • Create New...