Jump to content

Storage system in KVS tube script


Tech Support

Recommended Posts

KVS tube script has very simple and at the same time very powerful storage system, which provides the following benefits:

  • Unlimited storage space - add new servers / HDDs any time you need.
  • Support for traffic load balancing - can serve the same content from multiple servers at the same time.
  • Reliable protection from hotlink and unauthorized access.
  • Easy migration - admin panel can migrate content to another storage automatically.

Storage system settings are located in Settings -> Storage servers section in admin panel. Storage system is used to store video formats and album files (images and ZIPs).

NOTE: video screenshots are not stored in this storage system, they are stored locally with the main site. However there are some workarounds to move them to another server if needed, which is described in this article: How to move static files to CDN or another server in KVS

NOTE: since videos support 4 content types, only 1 content type is using KVS storage system: video upload. Other content types (hotlink, embed, pseudo) do not have any video files that need to be stored on your servers. When adding videos into KVS you should make sure you do not mix between video upload and hotlink, because both can be added using video file URL, but the difference is that video upload will process and store the downloaded file in KVS storage system (with all its benefits), while hotlink will be using the specified URL directly. You can read more about video content types in this article: What video types are supported in KVS tube script and how they are different

Though the concept is very simple and straightforward, many KVS users do not understand it and tend to use it in wrong way. Let us try to explain it in more details here.

NOTE: Storage systems for videos and albums work exactly in the same way. So we will only describe videos.

 

Storage groups and servers

When videos or albums are uploaded into KVS, they are stored not on storage servers directly, but on storage groups instead. A group can contain multiple storage servers, which is needed when you want to load balance the same content between multiple servers. We will say this in another way: if a group has multiple servers, ALL CONTENT of this group is DUPLICATED on all its servers even if some servers are deactivated. Each video of the group is stored on every server of this group. The common mistake when trying to move content to another server is to add the new server into the same group and disable the old server.

NOTE: disabling server in storage system doesn't mean that KVS won't copy content there. KVS will duplicate content between ALL servers added to the same group. Disabling a server means that KVS won't use this server when streaming videos to users, but it can be enabled back at any time and immediately it will start serving all the content stored on it.

So in most cases you should have 1 group = 1 server there. You should only add multiple servers into the same group in these cases:

  • You want to load balance streaming between 2 or more servers.
  • You want to have a backup server with all your content.
  • You want to manually move content from 1 server to another. But in this case after content migration you will need to delete old storage server and finally leave only 1 server in the group.

 

Sample diagram

Sample diagram shows all storage system concepts in 1 screen. Storage group 1 has 2 servers and you can see that both Video 1 and Video 2 are stored on both servers. Storage group 2 has only 1 server, which stores completely another set of videos: Video 3 and Video 4.

storage_system.png.6d065fdaabb4ea00eb57957316676232.png

Here is KVS configuration for this storage system:

storage_system_config.thumb.png.f5a963153b1775025ea919b4a833cd68.png

 

Adding more space to storage system

In order to expand your storage system you should create a new server group which will host new videos. Then you should add a new storage server into this new server group. That's it! In most cases new content will be copied to your new server group and thus new server under it.

However in Settings -> Content settings there is an option which configures default storage for videos (and same option for albums). By default this option is configured to autodetect group with the most free space. If you added a new server group with more free space but the new content is still being added to the old server group, most probably you have changed this option, so please check if it is set to autodetect:

storage_system_default_group.png.7923d2effa9463befd78a08506def532.png

 

Traffic load balancing between storage servers

Why would you need this in the era of CDN? Here are some common cases:

  • CDN can be more expensive than 2 standalone servers in US and Europe. So you can save by adding 2 servers instead and use GEOIP load balancing between them.
  • For CDN you usually pay for 90% traffic peak. You can reduce this payment by serving low-profile countries from standard server (use GEOIP load balancing) and thus reducing traffic usage of your CDN and reducing monthly payments.

In order to use load balancing you will need to add 2 (or more) servers into 1 group. If your storage group already hosts some videos, you will need to sync content on the new server by using sync option for it (this option is available on servers list). After using sync option KVS will create a background task for this operation which may take some time. If no content is hosted by your group, you don't need anything else, just add 2 servers into 1 group and configure their load balancing if needed in group settings:

storage_system_load_balancing.thumb.png.f552c1a0a9f72eba3a63f4ffaa87d39b.png

 

Moving content to another storage server

Add new server into the existing group and use sync option to copy all existing content to this server. Sync option will create background task and you have to wait for it to be completed. Then after you checked everything you can delete the old server from KVS and the group will have 1 new server.

NOTE: When a new server is added to a group which already hosts some content, this new server will be disabled by default, so KVS won't show it to users and you will have as much time as needed to manually copy all the content. Your site will still continue serving content from the old server until you manually activate the new server.

So basically here are the steps:

  1. Add new server into the existing server group.
  2. Sync new server with all the existing content.
  3. Wait for sync background task to be finished, it may take hours or days depending on how much content you have.
  4. Activate the new server, deactivate the old server.
  5. Verify that everything is working, your content is functional.
  6. Delete old server from server group.

storage_system_move.png.8a1e5f83c64cb3977f494fb657aa1d27.png

  • Like 1
Link to comment
Share on other sites

  • 5 months later...
  • 1 year later...
5 hours ago, Tanjiro said:

storage_system_load_balancing.thumb.png.f552c1a0a9f72eba3a63f4ffaa87d39b.png

I don't see this Load balancing page, where is it ? Could you please more specific

You can see it only when you have multiple server in the same server group. Then you can open server group editor and see this section to enable load balancing weights and countries.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...