-
Simple Bash Automation for EC2 Instance Backups using Cron on a Mac Mini
The problem that needed to be solved was to have a simple backup system that could be used to maintain a bunch of EBS backed Windows instances on AWS. We wanted to keep a weeks worth of backups, always discarding the older AMI’s. A local mac mini was used to run the necessary scripts and cron. We didn’t want just EBS snapshots, we wanted a full AMI everyday. There are other solutions if you just want to backup an EBS drive on a schedule, and even for this but this one suits our needs.
To install the ec2 api tools on the MAC mini we did the following:1. Launch terminal and create a folder called .ec2 in the users home folder:
mac-mini:~ User$ mkdir ~/.ec2
2. Download and extract the ec2 api tools from Amazon into the folder you just made. You only need the bin and lib folders.(https://aws.amazon.com/developertools/351).
3. Get your X.509 certificates or your Access Keys, I will go over using both methods. If you have more than one account Access Keys will be easier to use than the X.509 certificate. If you only have one account the X.509 is easiest. This link should take you to your security credentials (https://console.aws.amazon.com/iam/home?#security_credential). If using the X.509, copy/move the cert-xxxxxxx.pem and pk-xxxxxxx.pem files into the ~/.ec2 folder.
4. Edit your .bash_profile with the following to add environmental variables to make life easier:
# Setup Amazon EC2 Command-Line Tools export EC2_HOME=~/.ec2 export PATH=$PATH:$EC2_HOME/bin export JAVA_HOME="$(/usr/libexec/java_home)" # If using Access Keys you do not need the following: export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem` export EC2_CERT=`ls $EC2_HOME/cert-*.pem`
***IMPORTANT: IF YOU ARE GOING TO RUN THE SCRIPT IN CRON, MAKE SURE YOU ADD THESE TO THE SCRIPT AS WELL!!!
5. Quit and reload Terminal. You can now run the ec2 api tools from the command line.
6. Run a test to make sure you can access the tools:
mac-mini:~ User$ ec2-describe-instances
**if you are using Access Keys instead of X.509 you have to add the your identifiers when you issue a command:
mac-mini:~ User$ ec2-describe-instances -O YOURACCESSKEY -W YOURSECRETACCESSKEY
7. Launch terminal and create the bash script to make the backups. In this case we are making the script in a Scripts folder in the users home directory (you can switch vi for any other editor you prefer):
mac-mini:~ User$ vi ~/Scripts/ec2-backup.sh
8. Add the following into the script then save and close:
#!/bin/bash # Delete AMI's older than 7 days ec2-describe-images | grep `date -v -7d +%F` | awk '{print "Deregistering-> " $2; system("ec2-deregister " $2)}' # Create AMI's from existing instances without rebooting them ec2-describe-instances | grep instance | awk '{print "Creating -> " $3; system("ec2-create-image --name " $5 "-$(date +%F) --no-reboot " $3)}' # Delete old snapshots - NOTE THIS DELETES ALL SNAPSHOTS THAT ARE NOT ASSOCIATED TO AN AMI!!! ec2-describe-snapshots | sort -k 5 | awk '{print "Deleting-> " $2; system("ec2-delete-snapshot " $2)}'
This is what you would use if you are using Access Keys:
#!/bin/bash # Delete AMI's older than 7 days (must have the date in their name) ec2-describe-images -O YOURACCESSKEY -W YOURSECRETACCESSKEY | grep `date -v -7d +%F` | awk '{print "Deregistering-> " $2; system("ec2-deregister -O YOURACCESSKEY -W YOURSECRETACCESSKEY " $2)}' # Create AMI's from existing instances without rebooting them and putting the date next to the name ec2-describe-instances -O YOURACCESSKEY -W YOURSECRETACCESSKEY | grep instance | awk '{print "Creating -> " $3; system("ec2-create-image --name " $5 "-$(date +%F) --no-reboot -O YOURACCESSKEY -W YOURSECRETACCESSKEY " $3)}' # Delete old snapshots - NOTE THIS DELETES ALL SNAPSHOTS THAT ARE NOT ASSOCIATED TO AN AMI!!! ec2-describe-snapshots -O YOURACCESSKEY -W YOURSECRETACCESSKEY | sort -k 5 | awk '{print "Deleting-> " $2; system("ec2-delete-snapshot -O YOURACCESSKEY -W YOURSECRETACCESSKEY " $2)}'
*Your EC2 instance must be named, with NO spaces, for the AMI to be created with ec2-create-image using this script
** TO RUN SCRIPT FROM CRON, ADD ENVIRONMENTAL VARIABLES FROM STEP 4!9. Give your script execute privilege:
mac-mini:~ User$ chmod +x ~/Scripts/ec2-backup.sh
10. Add the script to crontab to run at 1am (system time) everyday:
mac-mini:~ User$ crontab -e #Backup script run at 1am 0 1 * * * /Users/User/Scripts/ec2-backup.sh
If you need any help, just leave me a comment and I will try to get you all sorted out.
**UPDATE**
With the update of the ec2 cli tools, there is now a new way to go about doing this using .json files. When time permits I will post how to make a cron ec2 backup job using the outputted .json file.
9 responses to “Simple Bash Automation for EC2 Instance Backups using Cron on a Mac Mini”
-
Hi,
I tried as you mention, but my cron job is failing with some error. it’s like “ec2-describe-images” and “ec2-describe-tags” not found. but thing is if run manually, it is working properly.. Could you please help me.
Thanks,
Naresh Ch -
Mudassir Aftab October 24th, 2013 at 09:21
Thanks for above script , can you describe following , will it terminate existing instances running more than a week ?
# Delete AMI’s older than 7 days (must have the date in their name)
ec2-describe-images -O YOURACCESSKEY -W YOURSECRETACCESSKEY | grep `date -v -7d +%F` | awk ‘{print “Deregistering-> ” $2; system(“ec2-deregister -O YOURACCESSKEY -W YOURSECRETACCESSKEY ” $2)}’
How can i exclude dev.xxx instances for backup ?
-
/opt/aws/bin/ec2-stop-instances: line 9: EC2_HOME: EC2_HOME is not set
WHY! I’ve been struggling with this problem for a couple of days. Cron is not running the aws commands. Keep getting emails that EC2-HOME is not set for the cron shell.
-
In my case it is giving error while executing
date: invalid option — ‘v’
Try ‘date –help’ for more information.I am using ubuntu
-
aws training June 17th, 2016 at 00:20
Oh my goodness! Incredible article dude! Many thanks,
Leave a reply
-
Naresh Chidipothu September 15th, 2013 at 01:34