Search the Community
Showing results for tags 'storage'.
KVS tube script has very simple and at the same time very powerful storage system, which provides the following benefits: Unlimited storage space - add new servers / HDDs any time you need. Support for traffic load balancing - can serve the same content from multiple servers at the same time. Reliable protection from hotlink and unauthorized access. Easy migration - admin panel can migrate content to another storage automatically. Storage system settings are located in Settings -> Storage servers section in admin panel. Storage system is used to store video formats and album files (images and ZIPs). NOTE: video screenshots are not stored in this storage system, they are stored locally with the main site. However there are some workarounds to move them to another server if needed, which is described in this article: How to move static files to CDN or another server in KVS NOTE: since videos support 4 content types, only 1 content type is using KVS storage system: video upload. Other content types (hotlink, embed, pseudo) do not have any video files that need to be stored on your servers. When adding videos into KVS you should make sure you do not mix between video upload and hotlink, because both can be added using video file URL, but the difference is that video upload will process and store the downloaded file in KVS storage system (with all its benefits), while hotlink will be using the specified URL directly. You can read more about video content types in this article: What video types are supported in KVS tube script and how they are different Though the concept is very simple and straightforward, many KVS users do not understand it and tend to use it in wrong way. Let us try to explain it in more details here. NOTE: Storage systems for videos and albums work exactly in the same way. So we will only describe videos. Storage groups and servers When videos or albums are uploaded into KVS, they are stored not on storage servers directly, but on storage groups instead. A group can contain multiple storage servers, which is needed when you want to load balance the same content between multiple servers. We will say this in another way: if a group has multiple servers, ALL CONTENT of this group is DUPLICATED on all its servers even if some servers are deactivated. Each video of the group is stored on every server of this group. The common mistake when trying to move content to another server is to add the new server into the same group and disable the old server. NOTE: disabling server in storage system doesn't mean that KVS won't copy content there. KVS will duplicate content between ALL servers added to the same group. Disabling a server means that KVS won't use this server when streaming videos to users, but it can be enabled back at any time and immediately it will start serving all the content stored on it. So in most cases you should have 1 group = 1 server there. You should only add multiple servers into the same group in these cases: You want to load balance streaming between 2 or more servers. You want to have a backup server with all your content. You want to manually move content from 1 server to another. But in this case after content migration you will need to delete old storage server and finally leave only 1 server in the group. Sample diagram Sample diagram shows all storage system concepts in 1 screen. Storage group 1 has 2 servers and you can see that both Video 1 and Video 2 are stored on both servers. Storage group 2 has only 1 server, which stores completely another set of videos: Video 3 and Video 4. Here is KVS configuration for this storage system: Adding more space to storage system In order to expand your storage system you should create a new server group which will host new videos. Then you should add a new storage server into this new server group. That's it! In most cases new content will be copied to your new server group and thus new server under it. However in Settings -> Content settings there is an option which configures default storage for videos (and same option for albums). By default this option is configured to autodetect group with the most free space. If you added a new server group with more free space but the new content is still being added to the old server group, most probably you have changed this option, so please check if it is set to autodetect: Traffic load balancing between storage servers Why would you need this in the era of CDN? Here are some common cases: CDN can be more expensive than 2 standalone servers in US and Europe. So you can save by adding 2 servers instead and use GEOIP load balancing between them. For CDN you usually pay for 90% traffic peak. You can reduce this payment by serving low-profile countries from standard server (use GEOIP load balancing) and thus reducing traffic usage of your CDN and reducing monthly payments. In order to use load balancing you will need to add 2 (or more) servers into 1 group. If your storage group already hosts some videos, you will need to sync content on the new server by using sync option for it (this option is available on servers list). After using sync option KVS will create a background task for this operation which may take some time. If no content is hosted by your group, you don't need anything else, just add 2 servers into 1 group and configure their load balancing if needed in group settings: Moving content to another storage server Add new server into the existing group and use sync option to copy all existing content to this server. Sync option will create background task and you have to wait for it to be completed. Then after you checked everything you can delete the old server from KVS and the group will have 1 new server. NOTE: When a new server is added to a group which already hosts some content, this new server will be disabled by default, so KVS won't show it to users and you will have as much time as needed to manually copy all the content. Your site will still continue serving content from the old server until you manually activate the new server. So basically here are the steps: Add new server into the existing server group. Sync new server with all the existing content. Wait for sync background task to be finished, it may take hours or days depending on how much content you have. Activate the new server, deactivate the old server. Verify that everything is working, your content is functional. Delete old server from server group.
By default KVS comes with local storage servers for videos and albums (if ultimate package). When you add a new storage server the first thing you should ask yourself is whether you want it to be in a new server group or in an existing server group. The difference is described in details in this article: Storage system in KVS tube script Here is short summary. New server group: You want to extend storage space. You want new content to be copied to new server, while old content should remain on old server. Existing server group: You want to load balance content traffic between multiple servers. You want to migrate existing content to a new server. NOTE: content servers for videos and albums work in exactly same way, so we will only describe that for videos. How to migrate your existing content to a new server or CDN The most common scenario is when you already have local storage and want to migrate to a remote server or CDN. In order to do that you should do the following steps: Step 1. Add a new content server into the same storage group where your old server is located. Don't worry, the new server will be added in Disabled status, so it won't affect your site users. You can find more info about adding new storage servers below. Step 2. Start content sync process for the new storage server. This process can be started in servers list in KVS admin panel. It will automatically copy all content stored in the group to a new server. NOTE: sync process will create a background task and you will be able to see its progress in the list of background tasks. Based on how many content you have, this task may take days and even weeks and it will be interrupted by processing new videos if any. If you want to manually copy content files via FTP, you can do that and skip this step. Step 3. When sync process is over, if possible verify the size of content folders on both servers to make sure they have the same size. At the same time verify that KVS is not showing any errors for the new server. There will be errors until all content is synced, but there should not be any errors afterwards. Step 4. Enable new server in storage group and disable the old one. Now your users will see content from the new server. Step 5. When you are sure there are no issues, delete old server. NOTE: KVS will stop copying new videos to the old server only after you delete it from KVS. Please make sure you remember where video files are stored on filesystem before deleting old server from KVS, see step 6 for more details. Step 6. When deleting server from KVS in step 5, KVS won't physically delete any video files there. Deleting a storage server from KVS only removes it from KVS records and doesn't affect anything on filesystem. This is done for security reasons, otherwise you could accidentally remove storage server from KVS and delete all your videos files. So the only way to delete video files from the old server (e.g. free the disk space) is to do that manually from server side. We recommend you to wait for 5-7 days after deleting old storage server from KVS so that you are 100% sure that all your site is working without any issues and all videos are playing fine. When you are sure everything is OK, you are good to delete old video files from server HDD. How to add a new content server System requirements for a content server in KVS: PHP 7.1+ No specific PHP modules required Apache and / or Nginx (recommended for better performance and content protection) NOTE: Apache is NOT strictly required for remote content server, you can have Nginx + PHP. You can add a content server in admin panel in Settings -> Storage servers. For server URL you should specify any URL that you want your content to be served from. It is a common practice to use IPs and subdomains here, as well as subfolders. It is not recommended to store content in www root folder, as this may bring some problems in future. So here is the recommended URL patterns: http://220.127.116.11/videos http://storage1.domain.com/videos http://storage2.domain.com/videos NOTE: if your site works via HTTPS, all your storage servers should also use HTTPS. This is why it is better to use subdomains here, as SSL certificates can support wildcard (*) subdomains and you can install the same certificate on all storage servers. Streaming type is an important option, which indicates the type of your storage server. If this server is external CDN provider, then you should select CDN (CDN configuration for videos and albums is a topic of another article, will not be covered here). If your server doesn't have nginx installed, then you should choose HTTP 302 redirect. This option means that KVS will redirect to video files directly without any protection. Video files which are available via direct links can be easily hotlinked by other sites and someone can be stealing your traffic. This is the common issue with wordpress and other scripts, they do not allow protecting your video files. KVS keeps this option for specific cases, but its usage is not recommended. In most cases your storage server should have nginx installed and in this case you should select nginx streaming type. The next set of options are related to server connection. KVS uses these settings to copy files to remote servers. In most cases you will need to specify FTP connection details here. The most confusing option here can be FTP folder. In order to specify it correctly you should first login to your FTP using standard FTP client and see where you come into. It can be possible that your FTP is directly configured to the needed folder, but in most cases FTP will be configured into home folder of FTP user or www root folder. Since your content will be stored somewhere in subfolder (e.g. http://storage1.domain.com/videos), you should specify RELATIVE path to this subfolder in this field. For example if your FTP is configured into www root (http://storage1.domain.com), then you need to specify: videos If your FTP is configured into user's home folder, then you will need to specify more subfolders to come to /videos: public_html/videos or www/videos or even data/domain.com/public_html/videos NOTE: each storage or conversion server in KVS should have unique folder. Do not configure different servers into the same folder on filesystem, this may result in data loss. Sometimes if you specify wrong FTP folder KVS will show error that it is not able to validate connection. However in many cases there will be no error, and the files will be copied to the wrong place and will not be available to the outside. Then after you add some videos to this server KVS will show an error on it, indicating that video files are not working from this server. Control script URL field is not editable. This field is built automatically from the URL field you specified above. It indicates where you should put KVS control script on this server, so that it is integrated with KVS. Control script serves 2 things: it provides info about server (its load and disk space) and implements protection for your content. There is a link near by to download this script and you should copy it manually to the storage server so that it is available at the displayed Control script URL. So if your storage server has this URL: http://storage1.domain.com/videos Then you should copy control script here: http://storage1.domain.com/remote_control.php You can manually run this script via browser and make sure it is working and returns "connected" word. Until this happens, you won't be able to save this server. Finally, there is Time offset field. In order to implement content protection KVS generates links which expire after some time (usually after 1 or 2 hours). In order to have this working, it is required that timezones on both servers are synced. If you don't want to configure the same timezone on your remote server, then you need to specify time offset, e.g. how many hours your storage server's timezone is behind or ahead in comparison to main server. However due to different daylight saving concepts in US and Europe it can be possible that after some time this difference will be shifted 1 hour back or forth. So the best way to ensure that remote server will not have any problem is to configure the same timezone on main server and remote server. Then you will not need to specify any time offset. Here is an example configuration for a remote storage server: Finalizing protection for nginx After the server is added, open it for editing. You may see that in server settings there is a textarea with nginx config displayed (if your Streaming type is set to nginx): You should add this piece of code into nginx config of your storage server to make sure that protected files are not available via direct links. After modifying nginx config you should restart nginx. There is no common guide on where do you find your nginx config. It depends on your server panel vendor. Most server panels do allow restarting nginx from panel GUI, but do not allow modify nginx config directly. You should figure this out with your host support. But if you do not configure this, your video files will still be available via direct links and no protection will be possible. CDN storage When adding CDN storage you should use CDN option in Streaming type field. Then you also need to put your CDN control script into /admin/cdn folder (create this folder as it doesn't exist by default), specify its name in CDN control script field and specify your secret key in Streaming key field: CDN control script is a PHP script that should be implemented by your CDN provider. This script implements content protection logic by creating temporary links to video or image files. Moreover, this script is responsible for invalidating content on CDN servers when files are replaced (modified) or deleted. NOTE: invalidation process may be very important for a tube site. Consider you have DMCA claim that requires you to delete copyrighted content. If invalidation is not working correctly, your CDN will continue to stream video files that should return 404 instead. Thus you may experience troubles with providing proof of deletion. Each CDN provider has its own API for protection and invalidation, there are no standards in this field. Therefore KVS requires that each provider implements their API and provides their CDN control script to customers for use in KVS. If no such script is provided, it will still be possible to use this CDN in KVS with default CDN control script, but no content protection or invalidation will be implemented and you will be missing these important features. CDN providers: if you are willing to implement CDN control script for your CDN, just ask customer to send you default CDN control script that can be downloaded in admin panel. This script provides a basic skeleton with all the needed functions having empty or basic implementation. If you want to use default CDN control script: Download default CDN control script and upload it to /admin/cdn folder. Specify cdnapi.php in CDN control script field. Specify any text in Streaming key field as this value is not used by default control script. Troubleshooting issues KVS will validate basic issues when you try to save server settings in admin panel. After you add server, KVS will validate it on background every 5 minutes to check if it operates successfully. Even if initially you didn't have any errors, new errors may appear later due to various factors. First, FTP connection or remote server configuration may be changed externally and some functionality will simply stop working. Also when you add a server which doesn't have any videos yet, KVS is not able to check if videos are streamed correctly or not. Only after you add some videos, this check will happen on background. All issues found on background are immediately reported to KVS start page as they all are considered CRITICAL and may indicate that your remote server is not working correctly and users are not able to see videos or photos. You should make sure to fix them ASAP. Errors from background checks are not very detailed and just indicate some set of issues. In order to get more details you should try to save server settings in admin panel. NOTE: if server reports an error and you have already fixed it, please give it 5 minutes to do re-check and verify this error is fixed. Until that, server may still show this error. This only applies for background checks on existing servers. When you save server settings in admin panel, all issues are validated right a way without any delay. If saving server in admin panel still shows validation errors, these errors are not yet fixed. Here is the list of errors you can get when saving server settings: - This connection folder is already used by another server. This issue means that you are trying to configure this server into a folder, which is already used by another storage or conversion server. You should never do this, each storage or conversion server in KVS should be configured into a separate folder on filesystem. - Unable to connect to host : post. KVS is not able to connect to the specified FTP host or port. If the entered data is valid, most probably your FTP is protected by firewall and you should make sure that your main server's IP is whitelisted. Also sometimes your FTP may be configured for using non standard FTP port, you should check this as well. - Unable to login with credentials provided. Either FTP username or password is not valid. - Put / get / chmod / delete operations failed, insufficient permissions possible. When validating server connection, KVS will try to put a test file to the server and then delete it. You will get this error if any of the mentioned operations failed, which probably indicates wrong FTP folder or FTP configuration issue. Try to log in to FTP manually using desktop FTP client and try to copy any file into the specified folder. Can you do this manually? Also this error can happen if /tmp folder on your main KVS installation does not exist or is not writable. In order to check that go to Plugins -> Audit and run installation check. It will show error for tmp folder if anything is wrong. - Control script failure. As we mentioned in this article, for remote storage servers you need to manually copy control script to make it available under the displayed Control script URL. You will get this error if KVS is not able to connect to it. Try opening control script URL in browser, do you see "connected" word printed? If you don't see it, then most probably you didn't copy control script to the mentioned location, or copied it to the wrong FTP location, or there is an issue with running PHP scripts on your storage server. If script works in browser but you still see this error, it can indicate that your main server is not able to connect to this URL due to DNS or other routing issue on your main server. You can check that by using server command line and typing the following command with your control script URL: curl -I http://storage1.domain.com/remote_control.php - Remote server time is not synchronized with primary server time. As we said already, it is recommended that PHP timezones on both servers are synced. You can get rid of this error by specifying timezone offset either positive (e.g. 2) or negative (e.g. -2). This error will show timestamps on both servers to give you clue how many hours you should specify. - This script is configured with another secret key, please update its secret key. Each remote control script has secret key hardcoded in it for protection. In some cases, specifically if you use multiple KVS installations and for each of them configure the same remote storage server, you can get this error. If you already had remote_control.php script on your storage server, then most probably it is already used by other KVS installation. If you indeed have other KVS configured with the same storage server, better contact support to make sure you don't harm another installation. If you modify secret key in remote_control.php and it turns out that this file is also used by another KVS, it will stop working there. Troubleshooting content check found errors The final validation that happens for every storage server is verifying that random videos can be played. If this check fails, you will see content check found errors issue displayed for server. You can further open content test page to see test logs, which include HTTP headers logs. Here are possible reasons: Your server doesn't have nginx installed but in KVS settings you set its streaming type as nginx. In this is true, then you need to change server's streaming type option to 302 redirect. Your nginx is not configured correctly, you should ask your host support to make sure that nginx configuration printed in KVS server settings was correctly added to nginx config for this server domain. CDN API script is not working as expected or the issue is at CDN provider's end. If your server is CDN, please open content test log for this server and send HTTP headers log to your CDN provider to describe the issue. Your Apache is configured with MultiViews option enabled, which prevents KVS from serving content correctly. In this case content test HTTP headers log will show 404 error. The requested file is missing on your server and thus returns 404 error which you can see in content test HTTP headers log. In most cases this happens because your FTP configuration has been changed externally and suddenly FTP user's home folder was changed to another one. KVS started copying new files to another location and of cause these new files are no more available under the old URL, configured in KVS server settings. You should login with FTP manually and check if its location is the same as what you have specified in server URL. If the location is different, then you should check with your host support where the correct location is and ask them to move content files from this new location to the old location and re-configure FTP user back to have home folder at the old location. For example your URL is configured as http://storage1.domain.com/videos and FTP access was configured directly into this folder. As time passed, KVS put many content into this folder but then suddenly your host support changed FTP access directly to domains www folder, e.g. this one: http://storage1.domain.com. The new files are no more copied under /videos folder, but copied under / folder instead. These new files will no more be available under http://storage1.domain.com/videos URL and KVS will report content check error.