Jump to content

How to use video grabbers more effectively


Tech Support
 Share

Recommended Posts

KVS features extremely powerful tube grabbers plugin (Plugins -> Grabbers). With this plugin you can import tons of content, even if you don't have any. We will not speculate on whether it is ethic or not to grab content from 3rd party sites, you should decide this yourself.

For basic understanding on how to use grabbers plugin please check the corresponding section in this post:

There are 3 different ways of how to import content from grabbers and as usual there is no perfect solution and each of them has its advantages and issues.

 

Grabbing as embed codes

This is the fastest way to fill your site with many content with few simple clicks. Install grabbers that support embed codes and set Mode of each grabber to Embed. Choose the other needed data you want to grab and submit lists where to grab videos from. Voila!

Pros:

  • The fastest way to import tons of content, as only screenshot will be download for each video.
  • No conversion power is required, can be used in virtually any server.
  • No storage and traffic payment is required, since no content is hosted within your server.

Cons:

  • You have no control on what is displayed inside embed code. The source site may opt to stop displaying embed codes, or to delete videos, or to redirect all visitors from embed codes to their site. You cannot control advertising that is displayed inside video player. But you can still use your advertising outside the player.
  • No support for video previews. The whole idea of using embed codes is to avoid video files being downloaded to your server. Therefore there is nothing to create video previews from.

See also: What video types are supported in KVS tube script and how they are different.

 

Downloading videos as a single source file

The idea here is that for each video KVS will download the file of the desired (typically maximum desired) quality, and then treat this file as a video source file to create all the needed formats from it (e.g. 720p, 480p, 1080p and video previews). In order to configure this you need to set grabber Mode to Download option and then choose which Quality you want to download as a source quality:

quality_single.png.b93a8ffe30021f4bcbf84c39b74add7b.png

So this option tells KVS that you want grabber to download 1080p file as a source file, and if a video doesn't have this quality - skip it. This will mean that only videos that have at least 1080p quality will be added by grabber. And if a video even has 4K quality, it will be added by grabber and KVS will download its 1080p file (not 4K file). This is normal, because you don't need to download 4K file, which has bigger size if your maximum video format is 1080p. Make sure you save your resources here.

The most important fact here is that the downloaded file will be treated as Source file with processing. This will mean that all video files will be fully created by KVS, and you can even configure to cut intros from them and add your own watermark.

Pros:

  • You control your videos and player, you can show any advertising that you want inside the player.
  • You do not depend on whether videos are removed from the source site or not.
  • You can cut intro rendered by source site, you can put your watermark on top of theirs.
  • Only 1 file needs to be downloaded, and everything else is created by KVS (but you still depend on the downloading speed for this file).

Cons:

  • Slower import speed than for embed codes, as you depend on the speed throttling imposed by source site.
  • Huge conversion power will be required to create all video formats.
  • Video files may have higher size after re-conversion, as KVS by default sets better compression quality then other sites. You may need to adjust compression quality, otherwise it may be a situation that the downloaded source file of 1080p quality is 100Mb, and KVS creates 720p quality (smaller resolution) that is 150Mb on disk - this makes absolutely no sense and should be checked and avoided.

 

Downloading videos as multiple ready-made files

This concept is different from the previous one in requiring KVS to download multiple video files of different qualities and uploading them as ready-made files without them being re-converted. In order to use it you need to set grabber Mode to Download and then choose Multiple Quality options similar to this:

quality_multiple.thumb.png.78e675760679d0684c6de62484d184ef.png

Configured like this KVS will download 480p, 720p and 1080p files (3 files in total) and will upload them into the corresponding video format slots. 240p and 2160p qualities will be skipped, even if source videos have them. The key fact here is that video files will be uploaded "as-is" without any conversion. This brings certain benefits and certain drawbacks.

Pros:

  • You still have full control on videos and player advertising in comparison to embed option.
  • And you don't need any processing power for this, so virtually any server will allow this.

Cons:

  • Since KVS needs to download multiple files, the importing speed will be much slower.
  • Since video files will be uploaded "as-is", you can't adjust them in any way, e.g. remove intros or do any watermarking.

 

How grabber download works in KVS and how to make it faster

Basically the last 2 scenarios are mostly different on what will be slower in your server environment: video conversion or downloading multiple files.

Where download limit comes from? It is normal that source sites don't want to lose their network channel on your project downloading their videos, so they typically impose speed throttling, either per download or per IP. In the first case when you download one file, this file's download speed is limited to 100 mbit/s, when you download another file in parallel, its download speed is limited to 100 mbit/s as well, but the summary of your download speed is now 200 mbit/s. In the latter case no matter how many downloads your start from the same IP, the summary download speed will be limited to 100 mbit/s.

So technically there are 2 ways of increasing download speed:

  1. Run downloads in parallel (will work if server only restricts speed per download).
  2. Run downloads in parallel + from different IPs (will work in all cases).

In KVS each grabber creates a separate import task that can itself be executed in multiple threads. You can configure the number of threads per grabber either in grabber settings (when you enable auto-pilot), or for the whole import, when you import set of lists manually.

So from the start if you grab videos via multiple grabbers, each grabber downloads its videos in a separate thread and thus you have multiple parallel downloads. You can see how this happens in Administration -> Background tasks -> filter by Status = In process:

multiple_import_processes.thumb.png.fc8613bfac0647d663033fecaf30895e.png

Here you can see 2 import tasks running in parallel. Each task comes from a separate grabber, each grabber has some download speed limit imposed by its source site, but the whole download speed of your system in increased. If you grab from 8 different sites, then the whole download speed will be ~8 times faster.

Then you can also configure multiple threads per each import task, for example 2 threads. This will mean that each import task will be executed in KVS in 2 separate threads and will simply grab different videos from the same source site. This will definitely work to increase download speed for sites that do not have speed throttling by IP. But even for sites that limit download by IP, it may help in some cases. The download limit is configured at streaming servers, and theoretically when you try to download different videos from the same site you may reach different streaming servers, and downloading from 2 different streaming servers using the same IP will not trigger any IP limit, as every streaming server has its own IP limit. In practice if you import random videos from huge sites, you will most likely reach different streaming servers.

So increasing the number of threads per grabber may also work to increase the overall download speed. However this comes with an increased risk of your server IP being blocked by source site, since the increased number of import threads will send an increased number of requests to their main site (not only to streaming servers). In order to reduce this risk we recommend to increase Timeout option available in each grabber settings. Timeout means the waiting interval for grabber to send next request to the source site. If your server sends 100s of requests within several minutes, its IP will be most likely banned, as no real user behaves like that. Therefore you should use some reasonable timeout time, which is set to 5 seconds by default, but should be definitely increased if you plan to grab videos 24/7. And each time you increase the number of threads per grabber, you also need to increase timeout so that the total number of requests within 1 hour or so is not increased.

Here are simple calculations. Let's say a grabber downloads 1 video in 10:00 minutes. And you have timeout set to 30 seconds. So the whole time for grabber to import a video is 10:30. If you increase the number of threads per grabber to 2, then assuming that each thread will download video the same 10 minutes, the average download speed will be reduced to 5 minutes per video. In order to keep the number of requests same, you will need to increase timeout twofold from 30 to 60. This means that each thread will import 1 video every 11 minutes, but having 2 threads in parallel results in average time per video equals to 11:00/2 = 5:30. If you increase the number of threads to 4 and increase timeout x4 to 120 seconds, then it will be 12:00 minutes (the full time for each thread to process and download video) / 4 threads = 3:00 minutes per video in average. That's much better!

And finally if you grab from 3 different sites in parallel you can increase your average importing speed to 1:00 minute per video, or 1440 videos per day. Even if downloading a single video takes 10:00 minutes. Of cause these calculations do not include some unexpected delays, when your site network is overloaded, or when source streaming server is overloaded and etc.

The importing speed actually means how fast videos are parsed and downloaded into your site. You can follow importing speed by looking into the number of new video tasks in Administration -> Background tasks section. If the number of new video tasks is small and doesn't grow with time, then importing speed is slower than your conversion engine can consume. You may need to think about further increasing the number of threads per grabber, or adding more grabbers for other sites. If the number of new video tasks is constantly growing, this means that your conversion engine is overloaded and makes no sense to increase importing speed anymore.

In the latter case you will need to understand how KVS conversion system works.

 

 

  • Thanks 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...