Back up your backups!
Backing up your files is important, but backing up your backups is important, too!
2021-02-10
Most people are using cloud storage providers as their file hosting solution. This makes sense because most people are storing simple file types like Word docs, Powerpoints, Spreadsheets, etc. and they are storing them in such small quantities that it does not incur any costs.
I believe Google Drive's free tier provides 15GiB of storage for free! This has been great for simple file storage. I use Google Drive to write Docs and Sheets that need to be highly available and easily shared. Google Drive is awesome for collaboration. I also use Google Drive to store archived data. Files that I don't need to use anymore, but I don't want to lose.
Reading and Writing
As you see above, the pricing for Google Drive is very reasonable. So, why don't I want to use it?
Well, the truth is that I would love to use it, but I want to be able to read and write to my cloud files as easily as I would interact with them locally. This is simple to do with Excel, Word docs, and other supported file types because we edit the files using their web editing tools, such as Google Docs and Google Sheets. Non-office files, however, are not as seamless.
Binary files or files that are not associated to an office suite are simply uploaded and downloaded from Google Drive. You cannot simply edit them in the web browser. This is not ideal as we must download the file, apply our changes, and then re-upload it to overwrite the old cloud copy.
It's possible to use the Google Drive sync software, but in my experience that software is not as customizable as I had hoped and it was slow when downloading/uploading files.
LAN Storage
I recently wrote a blog post about turning your Rasberry Pi into a Network Attached Storage device using Samba. Unfortunately, my external hard drive pooped the bed about 2 days after posting that blog.
I shouldn't be too surprised. This is the same hard drive that pooped the bed back in 2015, so I think this HDD is the problem.
I realized this meant I would have to either buy a new External hard drive, or maybe some USB to SATA adapters for some hard drives that I have lying around.
I came to the conclusion that it might be best if I simply buy a NAS instead.
I figure that purchasing a NAS will require less management from me than building my own. Also, using SATA rather than USB will provide faster read and write speeds to the HDD.
This means that the rasberry Pi can focus on hosting VPNs, DNS servers, and other micro services while the reading and writing is handled by the consumer-grade NAS device.
I'll let you guys know when I decide to buy one and what I based my decisions off of.
Use Case
I plan to set up the NAS with 2 Hard Drives. Ideally, the NAS will support RAID 1, so that we can prevent disasters such as a hard drive failure.
We'll split up the disk(s) with 2 partitions. One as an Archive partition and the other as a "working set" partition.
The archive partition's purpose is to store files that I don't plan to access for a long time.
We will also store device backups in that partition. These backups will be generated using duplicity.
Conversely, the other partition will be mounted to my devices as the set of files that I will be reading and writing to often. These will likely be assets and documents that are related to an active project.
I can access these files remotely thanks to the VPN server that I set up on my other blog post (read more here).
Back Up Your Backups!
Ah, disaster recovery.
Even if you back up your files, you gotta back up those backups!.
Our NAS will have protection against hard drive disasters via RAID 1. If you're unfamiliar with RAID 1, we're simply using 2 Hard Drives to act as 1. The second hard drive exists as a backup of the first.
The benefit of RAID 1 is disaster recovery and also faster HDD reading. This solution provides secure, reliable, and fast file access as well as locally managed backups.
The problem now is that all of our eggs are in one basket! if someone breaks or steals this NAS, all of our backups and shared files will be gone.
We've gone full circle here!
The final step is to back up our NAS to a cloud or offsite provider daily/weekly. I haven't looked into these providers yet, but I will perform some research to determine the most
cost-effective file archive/backup service. Technically, it doesn't even need to be a cloud service. As long as you are back uping to an off-site storage, that should be enough.
The cloud backup service does not need to be read or write quickly. Nor do I need frequent access to the files. It simply needs to be realiable and cheap.
We don't expect to use the cloud storage provider. It will only come into play if anything happens to the NAS.
If you have any questions about file backups and storage, please let me know.
Hope this was helpful!!