howto How to add remote conversion server in KVS

[Tech Support]

Multiple conversion servers can be used only in KVS ultimate package. They will let you reduce CPU load on main server and make video conversion faster, but only to some point. It is a common misunderstanding to assume that the more conversion servers you add, the more videos can be converted for the same time. This is not always true. The overall conversion speed depends not only on the actual conversion (e.g. video encoding), but on many other factors as well. Please read more about KVS conversion engine here.

 

Adding new conversion server in KVS

System requirements for a conversion server in KVS:

• PHP 7.1+
• IonCube Loader 10.2+ (not required if your have open source code option with your license)
• PHP Client URL Library (curl)
• PHP FTP
• PHP Image Processing and GD
• PHP.ini disable_functions should not contain "exec"
• FFmpeg 1.0+ with libx264, libavfilter and AAC codec (libfaac, libfdk_aac or native ffmpeg aac codec)
• ImageMagick

You can add a content server in admin panel in Settings -> Conversion servers.

For Maximum tasks option it is recommended to keep it in 5-10 interval. This option doesn't mean how many tasks are executed in parallel; instead it affects how many tasks are uploaded to this conversion server. High values for this option may result on overall conversion performance degradation. For Optimize content copying options we provided complete explanation in this post.

The next set of options are related to server connection. KVS uses these settings to copy tasks to conversion servers. In most cases you will need to specify FTP connection details here. The most confusing option here can be FTP folder. In order to specify it correctly you should first login to your FTP using standard FTP client and see where you come into. It can be possible that your FTP is directly configured to the needed folder, but in most cases FTP will be configured into home folder of FTP user or www root folder.

NOTE: conversion folder doesn't require any HTTP access to it. Therefore it can be ANY folder on your remote server, but it should be unique and should belong to only 1 conversion server in KVS.

Here is an example configuration for a remote conversion server:

 

Configuring cron on remote conversion server

NOTE: on conversion server with local connection you don't need to add any cron schedule for it, it will be automatically executed by KVS engine.

After conversion server is created in KVS and its connection is validated, KVS will copy remote_cron.php file into the specified remote folder. This file is needed to perform remote conversion tasks. In order for it to start working, you should put it on cron every minute using the following command (please note that you should specify the valid filesystem path to remote_cron.php script):

php /path/to/remote_cron.php > /dev/null 2>&1

Once configured, this script will create several files in the same folder, which will mean that this command works fine. KVS will change status for this conversion server from Initializing to Active within 5 minutes. After switching to Active status this server will be utilized by KVS conversion engine and will start processing conversion tasks.

 

Troubleshooting issues

KVS will validate basic issues when you try to save server settings in admin panel. After you add server, KVS will validate it on background every 5 minutes to check if it operates successfully. Even if initially you didn't have any errors, new errors may appear later due to various factors. First, FTP connection or remote server configuration may be changed externally and some functionality will simply stop working. All issues found on background are immediately reported to KVS start page as they all are considered CRITICAL and may indicate that your remote server is not working correctly and conversion may be blocked. You should make sure to fix them ASAP.

Errors from background checks are not very detailed and just indicate some set of issues. In order to get more details you should try to save server settings in admin panel.

NOTE: if server reports an error and you have already fixed it, please give it 5 minutes to do re-check and verify this error is fixed. Until that, server may still show this error. This only applies for background checks on existing servers. When you save server settings in admin panel, all issues are validated right a way without any delay. If saving server in admin panel still shows validation errors, these errors are not yet fixed.

Here is the list of errors you can get when saving server settings:

 

- This connection folder is already used by another server. This issue means that you are trying to configure this server into a folder, which is already used by another storage or conversion server. You should never do this, each storage or conversion server in KVS should be configured into a separate folder on filesystem.

 

- Unable to connect to host : post. KVS is not able to connect to the specified FTP host or port. If the entered data is valid, most probably your FTP is protected by firewall and you should make sure that your main server's IP is whitelisted. Also sometimes your FTP may be configured for using non standard FTP port, you should check this as well.

 

- Unable to login with credentials provided. Either FTP username or password is not valid.

 

- Put / get / chmod / delete operations failed, insufficient permissions possible. When validating server connection, KVS will try to put a test file to the server and then delete it. You will get this error if any of the mentioned operations failed, which probably indicates wrong FTP folder or FTP configuration issue. Try to log in to FTP manually using desktop FTP client and try to copy any file into the specified folder. Can you do this manually? Also this error can happen if /tmp folder on your main KVS installation does not exist or is not writable. In order to check that go to Plugins -> Audit and run installation check. It will show error for tmp folder if anything is wrong.

 

- No PHP extension for FTP was found. Your remote server's PHP doesn't have support for FTP module and therefore it is not possible to enable allow this server to copy content to storage servers option for this server.

 

- No PHP extension for CURL was found. Your remote server's PHP doesn't have support for CURL module and therefore it is not possible to enable allow this server to pull source files from primary server option for this server.

 

- Conversion script not configured / not working on this server.

- Conversion script is not working.

- Conversion script executed more than 15 minutes ago.

These errors indicate that your remote server either doesn't have a cron task to execute remote_cron.php script every minute, or executing this script results in critical PHP error. Please check if you have cron task set. If you have it, then try executing this command manually from command line to see its output (please note that you should specify the valid filesystem path to remote_cron.php script):

php /path/to/remote_cron.php

In some cases if your remote server has another timezone than your main server, KVS may think that its remote_cron.php script is not executed, while it does. Open conversion server settings in KVS admin panel and specify its timezone offset under Configuration. Save conversion server settings and give it 5 minutes to re-check and remove this error.

 

- Some libraries are not configured correctly on this server. This error indicates that KVS was not able to locate ffmpeg / imagemagick executables on your remote server. First of all you should make sure that these libraries are installed on the server. If not - please install them. If the libraries are installed, then most probably their binaries are not located using standard paths. Open conversion server settings in KVS admin panel and specify the correct paths for libraries that were not detected correctly under Configuration. Save conversion server settings and give it 5 minutes to re-check and remove this error.

[Dasweb]

Is there a way to configure ffmpeg to run different arguments on different conversion servers?

[Tech Support]

22 minutes ago, Dasweb said:

Is there a way to configure ffmpeg to run different arguments on different conversion servers?

This is not supported.

[Helmuts DNMUM.com]

Thank you for this information - we have just set up the 2nd conversion server, and it shows "Active".

--

My client has 1 Ultimate and 2x Basic + 2x Advanced KVS licenses that with time will all get upgraded to Advanced.

What process would you suggest to share conversion servers across several KVS sites?

Thank you,

Helmuts

Edited September 5, 2022 by Helmuts DNMUM.com

[Tech Support]

14 minutes ago, Helmuts DNMUM.com said:

What process would you suggest to share conversion servers across several KVS sites?

The only way to share the same physical server between multiple KVS projects is to separate them on filesystem, e.g. by creating different directories and configuring each directory as a separate conversion server in KVS.

[Helmuts DNMUM.com]

6 hours ago, Tech Support said:

The only way to share the same physical server between multiple KVS projects is to separate them on filesystem, e.g. by creating different directories and configuring each directory as a separate conversion server in KVS.

Perfect! Exactly what I thought. Thank you, Helmuts

[Tanjiro]

Hi,

I added a new conversion server (let's call it B) and the main server running the website (let's call it A).

Server B has higher specs than server A, so I want to set B as the primary conversion server. The local server (A) will be used as a backup or for conversions only when it has low CPU usage or when B has reached its maximum tasks, but B will remain the primary conversion server because it is stronger than A.

Can I set it up like that?

Edited May 25, 2024 by Tanjiro

[Tech Support]

Yes, you need to enable prioritize this server for new tasks option on server B, then it will grab all available tasks, and when it full other tasks will go to other conversion servers:

[Tanjiro]

I don't see it, is this feature belong to 6.2 version ?

 

 

Edited May 27, 2024 by Tanjiro

[Tech Support]

Yes, it is from 6.2.0:

5) In conversion servers you can now indicate if a server should be fully loaded with tasks before starting using other servers. This option may be useful if you have a more powerful conversion server together with some less powerful. In such a scenario if you don't activate this option, your conversion tasks may be randomly distributed to all servers, resulting in some tasks copied to less powerful servers, while more powerful will remain not fully loaded. Using the new option will result in faster processing, except for cases when you have big task queue waiting for processing and all your conversion resources should be loaded to the full extent.

[Tanjiro]

8 hours ago, Tech Support said:

Yes, it is from 6.2.0:

5) In conversion servers you can now indicate if a server should be fully loaded with tasks before starting using other servers. This option may be useful if you have a more powerful conversion server together with some less powerful. In such a scenario if you don't activate this option, your conversion tasks may be randomly distributed to all servers, resulting in some tasks copied to less powerful servers, while more powerful will remain not fully loaded. Using the new option will result in faster processing, except for cases when you have big task queue waiting for processing and all your conversion resources should be loaded to the full extent.

I'm still using php 7.4, is it okay to upgrade to latest version and next version upcoming ?

[Tech Support]

It is OK, KVS 6.2.1 still supported PHP 7.

[HakaVelli]

Quote

It is a common misunderstanding to assume that the more conversion servers you add, the more videos can be converted for the same time. This is not always true.

How many conversion servers would you suggest?

 

Quote

KVS conversion engine works in 3 phases, in some cases only 1 or 2 phases are executed and the rest are skipped. But if we speak of processing newly uploaded videos, this is always a 3-phase process:

1. Main server: prepares conversion task for conversion server and uploads this task to conversion server
2. Conversion server: executes task
3. Main server: verifies and finalizes task execution

 

Wouldn't it be faster and more efficient if both phases 1 and 2 were handled entirely on the conversion server, and only the finalized result was sent back to the main server?

[Tech Support]

11 hours ago, HakaVelli said:

How many conversion servers would you suggest?

If conversion server can download source files fast from your primary server, it may be 10-20 for example.

 

11 hours ago, HakaVelli said:

Wouldn't it be faster and more efficient if both phases 1 and 2 were handled entirely on the conversion server, and only the finalized result was sent back to the main server?

Not in all cases. Phase 1 needs access to database, and remote conversion will not have access to KVS database. Normally if you have a server and conversion servers in the same hosting, or at least same country, you won't have any issues with networking.

[HakaVelli]

On 12/6/2024 at 2:47 PM, Tech Support said:

If conversion server can download source files fast from your primary server, it may be 10-20 for example.

I didn't realize we could use that many conversion servers. I have a couple of questions:

1. Is it sufficient to use a VPS for some of these servers?
2. For some of the servers, would a simple dedicated server work well? For example, something like:
   • CPU: Intel Core i7-6700
   • RAM: 64 GB
   • Storage: 2 x 256 GB SSD
3. Can a server that I'm already using for image serving also handle conversions, since not all of the system's resources are being used at all times?
   @Tech Support

[Tech Support]

7 hours ago, HakaVelli said:

I have a couple of questions:

Yes to all questions. However the usage of low CPU is not good for conversion, it may be more beneficial to hire a top-grade CPU server, then multiple basic level servers.

[kukulia]

How do I assess the minimum disk space requirement for the conversion server?

Is it a function of

1) maximum upload size
2) maximum tasks?
3) other parameters?

[Tech Support]

Conversion server doesn't store any files and they are all temporary.

In order to estimate the disk usage by conversion could be this formula:

<max source file size> * 3 * <number of max tasks in conversion server settings>

For 4Gb source files and max 5 tasks per server it would be:
4Gb * 3 * 5 = 60Gb

In most cases it won't use even half of this, but in some cases it may fully process 5 tasks with creating all files and wait for the main server to clean the tasks.

[Mich]

10 hours ago, Tech Support said:

Conversion server doesn't store any files and they are all temporary.

In order to estimate the disk usage by conversion could be this formula:

<max source file size> * 3 * <number of max tasks in conversion server settings>

For 4Gb source files and max 5 tasks per server it would be:
4Gb * 3 * 5 = 60Gb

In most cases it won't use even half of this, but in some cases it may fully process 5 tasks with creating all files and wait for the main server to clean the tasks.

I’ve always wondered, what happens if disk space is missing?
How does the system react?
Let’s assume I’ve noticed that if the FTP server isn’t configured to accept enough simultaneous connections, KVS marks it in red but doesn’t provide much information for troubleshooting.

[Tech Support]

KVS doesn't check free disk space for conversion servers. I guess in case of disk space missing when copying files, the task will end up into error, and when new files are created by ffmpeg I'm not sure, most probably it should trigger error as well.