AWS: a simple backup suite

Posted: 03/03/2010 in Develop
Tags: , , ,

Hi All,
this release is about a small set of script that I use on our AWS server to make backup of data. Well, it is not complete cos I found every day something to add or modify, but for me is a nice point to start.

Two Words before start

Two words about my AWS server setup:usually, I use an ami on instance storage and two EBS volumes with same size.  Others details like Reserved IP are optional, cos any information about server is provided in the config files. I use to attach the two disk on /dev/sdo and /dev/sdq, patitioned with a single partition for each. If you don’t like, you have to check the script mkbackup.sh and fix.

The related mount point are /mnt/partition for /dev/sdo1 and /mnt/backup for /dev/sdq1, but remember this thing: only /mnt/partition is mounted when backup procedure is NOT running.

In /mnt/partition I move any folder that I want to backup: for example /srv/zope will be moved in /mnt/partition/srv/zope. You can see I respect the original tree of folders. On the root I create a symbolic link, so anyone can access to it without problems.

This is an example schema of my fs:

My backup suite is stored in /opt/backup_utils. Take care about path, because in config file there an explicit call to it.

Usually, we use monit to check the status of any application.

The Suite

The suite contains two tools: the first one make a copy of ami, the second one syncs the two EBS volumes and make snapshots. There’s a third tool to clean old snapshots, but is still in development. Soon will be available.

The first script is called amibackup.sh. It invokes in order: ec2-bundle-vol, ec2-upload-bundle and ec2-register. Calling this script is easy create a new ami starting from a running instance. It is usefull when bigs updated on server.

The second one is called mkbackup.sh and it is  more complicated. It is developed for daily backup and allows plugins. The flow of the actions is the follow: mounts backup disk, runs every enabled plugins, unmounts backup disk and call ec2-create-snapshot. The plugins make the rest.

At this time there are four plugins:

  • base: to sync folders but srv/zope without shutdown any service
  • mysql: an easy mysqldump
  • postgres: an easy pg_dump
  • zope: a complex solution for zope instance (with buildout or not, with zeoserver or not, with backup scripts or not)

A complete explain on zope plugin would be great but it is a little bit long.

Configure

The configuration files are two: excluded and conf. The first one contains a list of expression to identify which files will not copied: more in detail, is the parameter of –exclude-from option of rsync. The second one is the real configuration file, and it is a little bit long. It contains: AWS Info, Location Info, Db Info and Zope Info.

AWS Info: contains informations about the location of your AWS private key, AWS certificate, AWS user id, AWS access key, AWS secret key. These files and informations are provided by AWS. If do not know what they are, please check Quick Guide of EC2. Other informations in this section are: the bucket of S3 where you want to store your ami, the prefix filename of the ami, the region and the location where you are running your instance.

Location Info: base is the mount point of the backup disk, source_backup is the base folder for backup, rsync_options_base and rsync_options are the sets of parameters for rsync. Take care about rsync two options, because everyone is needed and used.

Db Info: contains login, password, and path to make a dump.

Zope Info: contains an important parameter called zope_scripts who contains the list of available script to start and stop zope. By default, I set instance (default buildout installation) and zopectl (default zope installation), but usually I add others with grok applications for example. The other parameter in this section is zope_pack_days: it means that the day we use to make the pack of ZoDB is sunday. Remember that this option works only if you have made a buildout with zeo/zope architecture.

Install

The setup is very easy: unpack the file in /opt/backup_utils and be sure to set the rights on file, usually I use chmod 700  *.sh and plugins too. After that you have to going in plugins-enabled, and make a symbolic link for every plugin you want. The idea is the same of apache virtualhost on debian systems or rc scripts.

After that, you can setup in cron the mkbackup.sh script to work daily.

Download

Click on me to download.

Licence

GPL v2

Advertisements
Comments
  1. […] my post AWS: a simple backup suite, I spent two words about what I mean with “persistent”. On AWS, if you use […]

  2. Bret Zhang says:

    Incredibly great read! Honest..

  3. […] details about that, please read this article. Cheers. Like this:LikeBe the first to like this […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s