Tech Support Posted October 11, 2017 Share Posted October 11, 2017 Multiple conversion servers can be used only in KVS ultimate package. They will let you reduce CPU load on main server and make video conversion faster, but only to some point. It is a common misunderstanding to assume that the more conversion servers you add, the more videos can be converted for the same time. This is not always true. The overall conversion speed depends not only on the actual conversion (e.g. video encoding), but on many other factors as well that will be explained in this post. KVS conversion engine works in 3 phases, in some cases only 1 or 2 phases are executed and the rest are skipped. But if we speak of processing newly uploaded videos, this is always a 3-phase process: Main server: prepares conversion task for conversion server and uploads this task to conversion server Conversion server: executes task Main server: verifies and finalizes task execution Main server cron and conversion server cron are both triggered each 1 minute. It means that any new video will require at least 2 minutes before its conversion is finalized. Even if conversion itself takes couple of seconds, you will have to wait for 2 minutes due to cron limits. Conversion phase 1 When phase 1 is executed, in most cases this phase is very fast and all it needs to do is to copy task info and the uploaded video file to conversion server. This may take a while if your source files are huge and your conversion server is remote. The most important thing to understand here is that this phase is always processed in 1 thread. If you have 10 new videos scheduled and 10 free conversion servers, the main cron will have to copy each video to each conversion server 1 by 1 (not in parallel). If copying each video takes 1 minute, your conversion engine will be busy with spending 10 minutes to copy 10 tasks to conversion servers. In this case there is no big difference how many conversion servers you have, because each task takes the whole minute to simply copy it. Your video queue will not be processed faster than 1 * 60 * 24 = 1440 videos per day. Remote conversion server settings have an option to cope with this issue, which allows conversion server to download video source files from the main server instead of require main server to copy them to conversion server: This changes the whole picture. Main server will not need to spend 1 minute to copy video source files anymore, instead it will spend only 1 second to copy a small task file. Thus it can copy 10 tasks to 10 conversion servers in less than 10 seconds, so that for the whole minute it can copy up to 60 tasks. Then each conversion server will download source files from main server, but now this will happen in parallel. If you have only 2 remote conversion servers, then they both will download 2 video source files at the same time. But if you have 10 - they will download 10 in parallel. This is a huge speed increase. The only issue here is to make sure that your main server is able to serve so many files at the same time, you should also remember that main server serves your website for the end users and these parallel downloads can affect the overall speed of your main server. Conversion phase 2 During phase 2 conversion servers will download video source from main server (if this option enabled in their settings, see the previous section) and will do all the necessary conversion logic programmed. Here each conversion server works independently of everything else. If you have 10 conversion servers, they will do phase 2 for 10 tasks in parallel, so phase 2 can be easily scaled to any speed you want by just adding more conversion servers. It is a common practice to create multiple conversion servers (at least 2) for the same physical server. You can easily do that by simply using separate conversion folders, so you can create 2 or more conversion servers in KVS using the same physical server box, e.g.: Convertor 188.8.131.52 - 1: /home/ftpuser/conversion1 Convertor 184.108.40.206 - 2: /home/ftpuser/conversion2 Convertor 220.127.116.11 - 3: /home/ftpuser/conversion3 Why does this make sense? The whole phase 2 conversion consists of multiple conversion steps, here are some examples of them: Downloading source file from the main server Using ffmpeg to convert video to another format or quality Using ffmpeg to take screenshots from video Using imagemagick to create thumbs for the screenshots Copying result video files to storage server(s) If you look at these steps, some of them are using much CPU and some of them are using much network to download or upload. Also when taking screenshots from video files ffmpeg is using less CPU in comparison to when converting, taking screenshots can result in some delays depending on source file and how easily it can be navigated based on keyframes. If you use only 1 conversion server for 1 physical server, there will be intervals when your CPU load is 0% or when your network load is 0%. This is a waste of resources. By creating at least 2 conversion servers for the same physical server you will increase its conversion speed and overall resources utilization will be close to 100%. But this doesn't mean that adding more conversion servers here will increase the speed to x2, or x3 - this is just a matter of +20% or +30%. Therefore we think that having 2 or max 3 conversion servers per 1 physical server is more than enough. Conversion phase 3 This final phase is executed back again on the main server. When conversion server finishes processing a task, main conversion engine will pull this task result from conversion server. All high-CPU operations have already been done on conversion server and this phase is just to pull result screenshots and result video files if needed. Copying files may take time, especially on low quality network connections. This phase is like phase 1 in term of time - it can take much time to copy big or many files and the number of conversion servers you have cannot speed this up. Similar to what we said in phase 1, phase 3 can also have optimized copying. Each conversion server has an option to enable it to copy video files to storage servers and thus main server will not have to do that: This will result in parallel video files copying and you can benefit here from having multiple conversion servers. There is one case however, when this option cannot be used - if you have a local storage server. The reason is that local storage server is defined as a directory path (e.g. /var/www/domain.com/contents/videos), and this directory makes sense for the main server only. Remote conversion servers cannot use this path, so they won't be able to copy result video files. In such cases KVS will automatically disable this option and will force all result video files to be copied via main server. Then you have no much sense to add many remote conversion servers, since they will probably have a lot of spare resources. Summary: what can affect conversion speed Here is a summary of factors that can affect conversion speed in different scenarios: Connection latency / network speed between main server and conversion servers - bad network conditions will reduce conversion speed, that's why remote conversion servers should be located ideally with the same datacenter to avoid network roundtrips. CPU and disk speed on conversion servers - converting big video files can utilize much CPU (when scaling, watermarking) and also disk subsystem. Very slow disks may result in CPU resources not fully utilized. RAM is not very significant, however we've seen some cases when ffmpeg or imagemagick was using more than 1GB of RAM. For a conversion server you should have at least 2GB of free RAM. If you use multiple conversion servers on the same physical server, then your physical server should have 2GB * number of conversion servers you plan there. In multiple remote conversion servers scenario if servers are allowed to download source files from main server, it is critically important that main server is able to serve all these files in parallel as well as it is able to serve your end users at the same time. Otherwise the benefit from using multiple conversion servers will be only in reducing CPU usage on the main server and not speeding up conversion process much. If you have any local storage server, which is used to host newly uploaded videos, then you will not get 100% benefit from using multiple remote conversion servers, as main server will have to pull all result video files from them in 1 thread, which will mean that your overall system conversion speed will be limited to the speed of pulling an average video file from a remote conversion server. You can still use 1 or 2 remote conversion servers for reducing main server's CPU usage. When uploading video files without conversion (e.g. you upload them not as source files, but as ready video formats, which is very popular when using tube scrapper plugin), CPU resources are mostly not being used. Thus in such scenarios the key component to speeding up new content processing is reducing network utilization. No much sense to use remote conversion servers, since no much CPU resources are being utilized. Instead remote conversion servers will increase network utilization and thus may reduce overall speed. If you are using local storage - you are fine with 1 local conversion server. But if you are using remote storage, you may need to add 1 more local conversion server (with another directory on filesystem), to parallel the process of copying video files to storage server(s) and thus to increase processing speed. How to analyze performance You can find conversion logs in Administration -> Background tasks log section: By looking into these stats you can have some understanding of what is the problem, if you have one. If most of the time is spent on creating video files, it may indicate that your CPU is quite slow. If much time is spent on copying or downloading files, it may indicate that your network connection is poor, or network is overloaded at some port (main, conversion or storage server), or disks are overloaded. Please pay attention to "Waiting for xxx to start phase Y" messages. They do not mean that conversion engine is stuck and "actively waiting", absolutely no - conversion engine may switch to another task during this period, which is the most likely scenario when you have many tasks. These waiting timings may be big enough, up to hours, but they are not considered when calculating "effective" time spent on task processing. Transcoding performance In most cases when you upload videos into KVS (not using embed codes or hotlinking 3rd-party video files) and your external storage / conversion servers do not have network bottlenecks, the major time is being spent on video transcoding. This is what background task log details names as "Creating format file". The transcoding is the process of decoding video file, resizing it to another size and compressing it back. This process uses a lot of CPU resources, and in some cases when your videos are in Full HD / 4K quality it may also utilize HDD due to big size of source and target files. Transcoding in KVS is typically used to create different resolutions of the same video, so that users can watch video in different qualities. Another reason of using video transcoding is to add watermark(s) on top of it. If transcoding takes much time in your setup and you are not happy with the number of videos that are being processed during each day, then there are few options to increase transcoding speed: 1) Add more external conversion servers so that more videos can be processed at once in parallel (possible with KVS Ultimate only). This is the most obvious, and the most expensive option. You need to buy some more server(s) for the time that you need to transcode your videos. 2) Use GPU-powered server and configure GPU conversion. GPUs are working faster on video transcoding due to various technical reasons, however the main issue with them is that you need vendor-specific ffmpeg codecs that will support transcoding on GPU. These codecs are never part of standard ffmpeg distributions (ffmpeg is the server library that does video transcoding), and they need to be built manually by your server support. Another problem is that you should change ffmpeg options in video format settings in KVS admin panel to get these video codecs working. 3) Avoid using transcoding and upload already prepared (transcoded) video files. This approach is typically used with grabbers and importing feeds, when the videos that you need to import are already transcoded into multiple qualities. All that KVS needs to do is to download them and put into storage system. No transcoding will happen in this case, so no CPU / GPU power is needed. However, your video files must be already present in all qualities you want to support, and also you will not be able to put any watermarks on them. You can find out more about uploading already transcoded video files in this article: If you are using grabbers, then check this article about speeding up grabbing: 4) Finally, if none of the above options works for you (you don't want to pay for conversion servers, or you are not using grabbers and don't have video transcoded in different video qualities), then the only way to avoid transcoding or reduce its time is to reduce the number of video qualities that your site offers. You can find them in Settings -> Video formats section of admin panel. By default KVS typically installs the following qualities: MP4 480p (the lowest and required) MP4 720p (for videos 720p+) MP4 1080p (for videos 1080p+) MP4 4K (for videos 4K+) For each of these video formats you can see the number of videos that have it. This can let you identify, if you win much time if you decide to stop using MP4 4K or MP4 1080p qualities for example. Are there many videos that have them in % to your videos that have MP4 480p? If yes, then you can increase transcoding speed in several times by simply disabling these formats. If most of your videos are 480p and 720p, then you won't win much from disabling them and have to do another choice: Delete 720p quality and keep all your videos in 480p. This would result in low visual quality and could become problem for your site in general. Actually 480p quality is too small for modern devices. Delete 720p quality and change 480p resize option to Do not resize (keep source size). This would result in each video will have the same quality as originally uploaded source file. No quality change will be possible, and all users will see video in the same quality. This could be good option, but you should keep in mind that if some of your source files are 1080p or 4k, they will be transcoded with the same size, and could take much disk space on your storage and utilize much network when somebody is watching them. So this option is good when you know for sure most of your source video files are of 720p / 480p quality; or you are OK with the fact that many videos will be served in 1080p+ quality from your servers. Also this option allows you completely avoid transcoding by uploading your source video not in Source file field, but in 480p format field (or better rename it to some other name to avoid confusion). Make 720p quality required (WARNING! this will create a set of tasks to create 720p for all videos that don't have it), and after it is created for all videos, delete 480p quality. So all your videos will be transcoded into 720p format only. If the source file is bigger than 720p (e.g. 1080p or 4K), its quality will be reduced to 720p; and if the source file is initially smaller - the quality will remain the same as source. This option is a good choice, when you want to serve decent video quality to your users, and you know that your source files have many of 1080p+, and you don't want to serve 1080p quality to all users. Quote Link to comment Share on other sites More sharing options...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.