I download lots of media files. So far I have been storing these files after I am done with them on a 2TB hard disk. I have been copying over the files with rsync. This has so far worked fairly well. However the hard disk I am using is starting to get close to full. Now I am needing to find a solution so I can span my files over multiple disks. If I were to continue to do it as I do now, I would end up copying over files that would already be on the other disk. Does the datahoading community have any solutions to this?
For more information, my system is using Linux. The 2TB drive is formatted with ext4. When I make the backup to the drive I use ’rsync -rutp’. I don’t use multiple disks at the same time due to having only one usb sata enclosure for 3 1/2 inch disks. I don’t keep the drive connected all the time due to not needing it all the time. I keep local copies until I am done with the files (and they are backed up).
If you have the space and budget, you should buy a cheap 4-bay NAS and use that as a backup drive over the network. That will remove a lot of the manual effort needed for a backup, offer future expandability and support raid options that protect the data against a hard drive failure.
Alternatively, you could have a second drive and use LVM to create a partition that extends across both drives, so it I’ll show up as one big drive that you can rsync to. The downsides are that you’ll need both drives attached to be able to mount it, and if either drive dies, you won’t be able to access the data.