# Backup

The organic design backups are created daily and compressed to 7zip and distributed over Secure Copy Protocol (SCP) to various other domains.

## ADrive - Excellent Cloud Storage Service

There's a lot of very good backup services around now, many offering 50GB or more of free storage space, but one thing I haven't been able to find for free is a service offering a good amount of space that allows files to be transferred using standard Linux shell commands such as rsync and scp. Last month I found ADrive which offers an excellent 100GB service for only $2.50 per month that has many useful features such as public files, shared online editing and more. But most importantly from my perspective is that they allow transfer via rsync and scp with zero-configuration and an excellent data transfer rate! this means I can have my servers automatically transfer my encrypted backup files to the ADrive cloud storage on a schedule. Sign up with my referrer link and then check out our rsync article for details about how to automatically sync encrypted backups to your ADrive account. ## Setting up automated backups over SCP The backups are done by running the backup-host.pl script from root. The script expects each host in the backup network to have a dedicated user called scp and a directory called /backup which must be writable by the scp user so that backups being transferred from remote servers can be put there. We can create the directory and give it the minimum level of access as follows, mkdir /backup chown root:scp /backup chmod 730 /backup This allows the scp user to create new backup files and write to them, but it doesn't have any permission to read them. Each scp user will have the same RSA key pair and authorized_keys file in it's .ssh directory, the latter containing the following line to allow the scp user to do nothing but accept files into the /backup directory (sending is done by the root user, so only receiving needs to be allowed by the scp user). command="scp -t /backup",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAAAB3Nza...jdIKh4jjd scp@od When initially setting up a new host to be part of the backup network a manual transfer must first be done to each of the remote hosts to ensure their fingerprints are added to the known_hosts file. The target directory is just "/" because the root directory for scp users is forced to /backup in their authorized_keys file. This must be done from a root shell because the scp users don't have permission to do anything except receive backup files. echo "TestFile" > foo scp -i /home/scp/.ssh/id_rsa foo scp@host1.com:/ scp -i /home/scp/.ssh/id_rsa foo scp@host2.com:/ scp -i /home/scp/.ssh/id_rsa foo scp@host3.com:/ ### The configuration file The configiration for the backups is done via a file in the same directory as the script called backup-host.conf The following parameters are recognised: ####$admin

The email address that important information such as low space or failed transfers should be sent to. The default is admin@organicdesign.co.nz.

#### $host The name that should be included at the start of the backup files to identify the machine that they're backed up on. The default is the machines hostname, but it's useful to be able to override this since many servers are given meaningless hostnames, or have the same hostname as other servers in the network. ####$pass

The MySQL root password for backup up the MySQL databases, and/or for encrypting the backups of confgiuration files.

#### $disk This is the device name such as /dev/sda1 that, if set, will report low space notification to the admin email address so that manual file pruning can be done if necessary. ####$free

The number of gigabytes of free space below which a low-space notification email will be sent to the admin email address.

#### @conf

A list of files that should be encrypted (with root MySQL password) as they may contain sensitive information such as passwords or private keys.

#### @files

A list of file/directory locations to include in weekly file backups.

#### @excl

A list of file/directory locations that should be excluded from the file backups, this should include locations which are very large and don't undergo change since they'll be better manually backed up, and also any files that may contain sensitive information. Note that there's no need to include any of the locations listed in @conf as these will automatically be excluded.

#### @scp

A list of servers to send backups to over SCP protocol using the scp user.

### File pruning

The backup files will take up a huge amount of space as time goes on, because each year they'll be 365 database backups and 52 file-system and configuration backups, and there could be a number of hosts sending their backups to each server too. To prevent the space getting quickly consumed, the backup files are "pruned" so that there are fewer and fewer of them as they get older.

There are only half as many files after they're older than a month, a quarter over two months, and so on. After a year there is only one file per month, and only one file per year more than two years old.

Another way of reducing the space is by not including any standard code-bases etc in the backups - anything that will be rebuilt by going through the documentation for the server has no reason to be backed up. Only parts of the system which are not covered by documentation such as the mathaba.net main site need to have a full file-system backup, but even this may only need to be a one-off backup with just minimal areas of the file-system being regularly backed up - such as locations to which files are uploaded etc.

## Wiki backups

We use a general backup script to backup wikis on the various servers we administer which is in our Subversion tools repository here. The script dumps databases and compresses them to 7zip, then sends them over SCP to a remote server. The script takes two parameters, the first is the filesystem path to the wiki, and the second is the SCP target location. The script obtains the database connection details directly from the wiki's LocalSettings.php file. Here's a snippet of Perl code from the script showing the exporting, compression and remote connection:

qx( mysqldump -u $wgDBuser --password='$wgDBpass' --default-character-set=latin1 -A > $sql ); qx( tar -r -f$tar $sql ); qx( 7za a$tar.7z $tar ); qx( chmod 640$tar.7z );
qx( rm $sql$tar );

qx( scp $tar.7z$scp );

## Online backup services

• Tarsnap - very cost-effective if you don't have a lot of data
• ADrive - very cost-effective if you do have a lot of data
• Dropbox - very cost-effective for space, but non-trivial connectivity