How do you backup your KVS servers and which solutions do you use?

[Helmuts DNMUM.com]

Good morning,

Simple question: How do you backup your KVS servers and which solutions do you use?

Acronis is great, of course, though really expensive.

Also, do you backup your conversion servers?

It would be great to hear what price-effective solutions you've found.

Best wishes!

Helmuts

[Mich]

Hi,

The approach I adopt for backups varies across my projects.

For my largest project to date, I've developed a relatively straightforward, efficient, and cost-effective backup strategy. This involves utilizing a custom bash script with Borg, and storing the backups on a Hetzner storage box.

For smaller projects and those external to KVS, when the data volume isn't excessive, I prefer Duplicity combined with S3. This also involves a custom bash script, adapted to meet my requirements, initially sourced from here.

An alternative for non-technical users seeking a more straightforward, albeit pricier solution, is Snapshooter.

The items I back up include my websites and one of my primary CDNs. I do not back up conversion servers, but I've developed my own Ansible Playbook and Docker Image for rapid deployment when necessary.

I understand the new KVS version will support external storage for backup. However, I will likely not utilize it due to its lack of SFTP support. While S3 is great, it can be quite costly. I've considered creating my own S3 equivalent, but it has not yet been justifiable.

In my view, executing a proper backup is one of the most challenging tasks.

I strive to optimize my backups for efficiency and cost. This implies maximum compression, rapid processing (aided by effective compression and security algorithms), and minimal cost. For example, Hetzner provides 5TB of storage for only 10 EUR.

What I often find lacking is user-friendliness.

Although files can be quickly and easily located, this requires a basic understanding of the backup package in use. Bash scripts can extract the backup and unarchive it, but using SSH and having minimal Linux knowledge is still necessary.

I hope this information is helpful. Keep in mind that there's no one-size-fits-all solution; the best backup strategy depends on your specific infrastructure and budget.

[naughtyking]

On 6/9/2023 at 2:03 AM, Mich said:

Hi,

The approach I adopt for backups varies across my projects.

For my largest project to date, I've developed a relatively straightforward, efficient, and cost-effective backup strategy. This involves utilizing a custom bash script with Borg, and storing the backups on a Hetzner storage box.

For smaller projects and those external to KVS, when the data volume isn't excessive, I prefer Duplicity combined with S3. This also involves a custom bash script, adapted to meet my requirements, initially sourced from here.

An alternative for non-technical users seeking a more straightforward, albeit pricier solution, is Snapshooter.

The items I back up include my websites and one of my primary CDNs. I do not back up conversion servers, but I've developed my own Ansible Playbook and Docker Image for rapid deployment when necessary.

I understand the new KVS version will support external storage for backup. However, I will likely not utilize it due to its lack of SFTP support. While S3 is great, it can be quite costly. I've considered creating my own S3 equivalent, but it has not yet been justifiable.

In my view, executing a proper backup is one of the most challenging tasks.

I strive to optimize my backups for efficiency and cost. This implies maximum compression, rapid processing (aided by effective compression and security algorithms), and minimal cost. For example, Hetzner provides 5TB of storage for only 10 EUR.

What I often find lacking is user-friendliness.

Although files can be quickly and easily located, this requires a basic understanding of the backup package in use. Bash scripts can extract the backup and unarchive it, but using SSH and having minimal Linux knowledge is still necessary.

I hope this information is helpful. Keep in mind that there's no one-size-fits-all solution; the best backup strategy depends on your specific infrastructure and budget.

why you just not buy second server and copy the first server to second server? you can use HETZNER SERVER AUCTION .

[Mich]

On 6/16/2023 at 7:01 PM, naughtyking said:

why you just not buy second server and copy the first server to second server? you can use HETZNER SERVER AUCTION .

Hi, have you considered what you might do with the surplus RAM, CPU, or network resources? Additionally, it's worth noting that this approach could incur significantly higher costs compared to the other solutions that I've suggested.

I fail to see the justification for deploying a secondary server, especially one with excessive resources, solely for the purpose of backing up files.

[hbarnetworks]

On 6/17/2023 at 1:01 AM, naughtyking said:

why you just not buy second server and copy the first server to second server? you can use HETZNER SERVER AUCTION .

Fyi hetzner does not allow any form of adult content.

[naughtyking]

On 6/18/2023 at 2:30 AM, Mich said:

Hi, have you considered what you might do with the surplus RAM, CPU, or network resources? Additionally, it's worth noting that this approach could incur significantly higher costs compared to the other solutions that I've suggested.

I fail to see the justification for deploying a secondary server, especially one with excessive resources, solely for the purpose of backing up files.

i really dont know what is Hetzner storage box. or Snapshooter. 
i know that is there two way for backup, one dropbox, but its not recoomend becaouse if you website fall, its will take days and maybe weeks to bring it back, its take time to transfer files from dropbox to your dedicated server if you have big content data like me (20TB).

and other way to backup is second server that can replace the first server if it fall.

can you tell me about snapshooter and hetzner storage box? how its work? is it good idea to backup 30TB of data on it?

[Mich]

On 6/19/2023 at 10:10 AM, naughtyking said:

i really dont know what is Hetzner storage box. or Snapshooter. 
i know that is there two way for backup, one dropbox, but its not recoomend becaouse if you website fall, its will take days and maybe weeks to bring it back, its take time to transfer files from dropbox to your dedicated server if you have big content data like me (20TB).

and other way to backup is second server that can replace the first server if it fall.

can you tell me about snapshooter and hetzner storage box? how its work? is it good idea to backup 30TB of data on it?

In a backup scenario, the second server will not take over the load from the first one. This is a much more complex task. For a low-cost, large-scale backup solution, you can opt for a dedicated server specialized in storage.

You can use Duplicity with Dropbox, however, this is not something I would recommend for this size despite it possibly working. Please note that I am unaware of Dropbox's bandwidth limitations. Restoration could take a lot of time, unless you have your storage separated across multiple servers and have only lost a few hundred GBs rather than TBs.

SnapShooter is a backup tool (graphical). For cost considerations, I would recommend opting for Dedicated Storage over Storage Boxes. Especially if you're going to have more than 20TB in the very near future. Otherwise, a storage box could serve your needs for now.

https://www.hetzner.com/dedicated-rootserver/matrix-sx

[hbarnetworks]

On 6/19/2023 at 4:10 PM, naughtyking said:

i really dont know what is Hetzner storage box. or Snapshooter. 
i know that is there two way for backup, one dropbox, but its not recoomend becaouse if you website fall, its will take days and maybe weeks to bring it back, its take time to transfer files from dropbox to your dedicated server if you have big content data like me (20TB).

and other way to backup is second server that can replace the first server if it fall.

can you tell me about snapshooter and hetzner storage box? how its work? is it good idea to backup 30TB of data on it?

You don't really need to backup your content. This seems like an impossible task. Having multiple storage boxes per group 2+ in different datacenters is the way to go.

[Helmuts DNMUM.com]

On 6/22/2023 at 3:29 PM, hbarnetworks said:

You don't really need to backup your content.

Thank you for all your suggestions. Could you please go a bit more on this one?

--

.. I am actually thinking of a solution > RClone + Storj

[Mich]

On 6/28/2023 at 10:30 AM, Helmuts DNMUM.com said:

Thank you for all your suggestions. Could you please go a bit more on this one?

--

.. I am actually thinking of a solution > RClone + Storj

I wanted to set up my own storj node a long time ago for passive income, but I never had the time, lol, and it was only worth it in the second or third year if you were lucky and had a sizable disk.

It seems like a great solution, in addition to being original. However, if you're not willing to pay for egress, there are less expensive options available. And it's always cheaper to opt for a large dedicated Hetzner server for storage (if the need is there, of course).

Storj has the same advantages as S3, and maybe even better durability and performance, although that doesn't matter much for backups.

I'm not sure how reliable their service actually is; the prices seem reasonable but could be lower. What really puzzles me is the "*Additional per-segment fee of $0.0000088 applies." If you're using rclone, this should be taken into account. I've never dealt with segmentation on rclone (rclone config set --drive-chunk-size 64M), but to summarize, you need to make sure it's properly configured to minimize segmentation and avoid being overcharged.

Here's an interesting post about segmentation: https://forum.storj.io/t/question-about-segments-and-cold-storage/19726/2

[Universal]

I use Syncthing to back up all files (videos and website). Maybe this will be useful for you. It's free and works pretty fast.

[hbarnetworks]

On 6/28/2023 at 4:30 PM, Helmuts DNMUM.com said:

Thank you for all your suggestions. Could you please go a bit more on this one?

--

.. I am actually thinking of a solution > RClone + Storj

If you have 3 different storage servers as an example with 3 different datacenters. I dont really see any reason for a backup. If your site is more personal than ofcourse you need backups. But for a P*rn site I cannot backup 200tb of data somewhere.