howto How to use Amazon S3 in KVS for storage / streaming and backup storage

[Tech Support]

Before using Amazon S3 or any S3 services please consider the pricing options, as they can go up to unreasonable amounts, because the services may charge you based on the number of requests / the amount of data stored. KVS will send ping requests each 5 minutes, so even if you don't have any traffic or storage server is deactivated, there will be some number of requests per month done by KVS. According to Amazon pricing, 1000 of requests would cost around $0.005, so the overall amount taken by utility requests would be very low. However Amazon Free Tier option for S3 is limited to 20000 requests per month, and KVS would exceed this number.

 

Creating Amazon S3 Bucket

You should create separate buckets for videos and photos, because their access levels are different. Videos should be protected from accessing by direct URLs, while photos normally should not. This section will explain for videos, and then we will summarize the difference that needs to be done in Amazon S3 settings for photos.

If your videos are divided into multiple storage groups, you may have the same video bucket for them, but you must then specify different S3 subfolder in KVS settings to make sure files from different storage groups are not mixed up in the same structure. Same for photos.

Go to Amazon S3 management console and create a Bucket. Specify the following fields (anything not mentioned keep by default):

• Bucket name: unique name for your bucket, e.g. myawsbucket123.
• AWS Region: choose the region where you want your data to be stored, e.g. US East (N. Virginia)
• Block Public Access settings for this bucket: switch OFF Block All public access option. Then agree with "I acknowledge that the current settings might result in this bucket and the objects within becoming public." warning.

Then save and if there are no errors from Amazon, your bucket will be created.

 

Acquiring S3 access keys

On the top of S3 Console GUI click your username and choose Security credentials option in user menu. Scroll down to Access keys and click Create access key button. After creating an access key, copy its Access key and Secret access key values locally, as you can only see this info once. If at any point you lose this, you can create a new one.

Note: for security considerations Amazon recommends creating a separate IAM user and use its access keys instead of using access keys of root user. Please refer to Amazon manuals if you want to do that. In KVS you will have to store your access keys in open format, so anyone who can access your server will be able to see your Amazon access key.

 

Creating storage server in KVS

When you add a new storage server the first thing you should ask yourself is whether you want it to be in a new server group or in an existing server group. The difference is described in details in this article: Storage system in KVS tube script

Here is short summary.

New server group:

• You want to extend storage space.
• You want new content to be copied to new server, while old content should remain on old server.

Existing server group:

• You want to load balance content traffic between multiple servers.
• You want to migrate existing content to a new server.

 

Step 1. Use FTP client or File manager of your server to copy /admin/tools/cdnapi_aws.php file into /admin/cdn/cdnapi_aws.php file (/admin/cdn directory doesn't exist by default, so you also need to create it). Modify the contents of the copied file and specify your access key and your secret access key in the corresponding fields on top. By default it will create URLs expired in 60 minutes, if you plan to have longer videos you need to adjust this value to higher number.

 

Step 2. Go to KVS admin panel -> Settings -> Storage servers -> Add storage server GUI. Specify fields as the following:

• Title: server name of your choice.
• Storage group: either existing or a new storage group.
• Streaming type: choose CDN if you plan to stream videos from this server, or choose No public access if this is just a backup server.
• URL (if you used CDN streaming type):

  https://<your_bucket_id>.s3.<your_bucket_region_id>.amazonaws.com

  Replace <your_bucket_id> and <your_bucket_region_id> with the data fro AWS Console, you can see that in Properties tab of bucket. In our case this would look like:

  https://myawsbucket123.s3.us-east-1.amazonaws.com

  NOTE: if you plan to store files in some subdirectory, also add it into the URL, e.g.:

  https://myawsbucket123.s3.us-east-1.amazonaws.com/subdirectory

• CDN control script (if you used CDN streaming type): cdnapi_aws.php
• Streaming key: doesn't matter, you can specify test.
• Connection type: S3-compatible.
• S3 region name: same region ID you used in the URL, e.g. us-east-1
• S3 endpoint URL: leave empty.
• S3 bucket name: same bucket name you used in the URL, e.g. myawsbucket123
• S3 subfolder: leave empty if data is stored in root, specify relative path if you plan to store data in subfolder, e.g. subdirectory
• S3 access key: specify your access key.
• S3 access key secret: specify your access secret key.
• S3 upload chunk size: leave empty.

Now save. If you have some validation error on save saying that not possible to put / get files, re-check if your connection fields are correct and you correctly copied access keys.

 

Step 3. If you added this server into existing group with some content, you need to sync it. Go to Storage server list in KVS admin panel and use Sync content option on this storage server. This should create a background task for syncing all content from the group to it. When the sync task is finished (you can track its progress in the list of background tasks), use Test content serving option for this server. If there are not errors reported, you can activate the server and it will start serving to public.

 

Different settings for albums storage

Albums normally need public access, so in addition to all the above steps you need to add public access policy in the bucket security settings for your albums bucket. Got to Amazon S3 console, open albums bucket and switch to its Permissions tab. Under Bucket policy click Edit button and specify the following policy code:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your_bucket_id>/*"
        }
    ]
}

Replace <your_bucket_id> with the ID you used for S3 bucket name field in KVS.