• Simple Bash Automation for EC2 Instance Backups using Cron on a Mac Mini

    The problem that needed to be solved was to have a simple backup system that could be used to maintain a bunch of EBS backed Windows instances on AWS. We wanted to keep a weeks worth of backups, always discarding the older AMI’s. A local mac mini was used to run the necessary scripts and cron. We didn’t want just EBS snapshots, we wanted a full AMI everyday. There are other solutions if you just want to backup an EBS drive on a schedule, and even for this but this one suits our needs.


    To install the ec2 api tools on the MAC mini we did the following:

    1. Launch terminal and create a folder called .ec2 in the users home folder:

    
mac-mini:~ User$ mkdir ~/.ec2

    2. Download and extract the ec2 api tools from Amazon into the folder you just made. You only need the bin and lib folders.(https://aws.amazon.com/developertools/351).

    3. Get your X.509 certificates or your Access Keys, I will go over using both methods. If you have more than one account Access Keys will be easier to use than the X.509 certificate. If you only have one account the X.509 is easiest. This link should take you to your security credentials (https://console.aws.amazon.com/iam/home?#security_credential). If using the X.509, copy/move the cert-xxxxxxx.pem and pk-xxxxxxx.pem files into the ~/.ec2 folder.

    4. Edit your .bash_profile with the following to add environmental variables to make life easier:

    
# Setup Amazon EC2 Command-Line Tools
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin
export JAVA_HOME="$(/usr/libexec/java_home)"
# If using Access Keys you do not need the following:
export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem`
export EC2_CERT=`ls $EC2_HOME/cert-*.pem`

    ***IMPORTANT: IF YOU ARE GOING TO RUN THE SCRIPT IN CRON, MAKE SURE YOU ADD THESE TO THE SCRIPT AS WELL!!!




    5. Quit and reload Terminal. You can now run the ec2 api tools from the command line.

    6. Run a test to make sure you can access the tools:

    
mac-mini:~ User$ ec2-describe-instances

    **if you are using Access Keys instead of X.509 you have to add the your identifiers when you issue a command:

    
mac-mini:~ User$ ec2-describe-instances -O YOURACCESSKEY -W YOURSECRETACCESSKEY

    7. Launch terminal and create the bash script to make the backups. In this case we are making the script in a Scripts folder in the users home directory (you can switch vi for any other editor you prefer):

    
mac-mini:~ User$ vi ~/Scripts/ec2-backup.sh

    8. Add the following into the script then save and close:

    
#!/bin/bash
# Delete AMI's older than 7 days
ec2-describe-images | grep `date -v -7d +%F` | awk '{print "Deregistering-> " $2; system("ec2-deregister " $2)}'
# Create AMI's from existing instances without rebooting them
ec2-describe-instances | grep instance | awk '{print "Creating -> " $3; system("ec2-create-image --name " $5 "-$(date +%F) --no-reboot " $3)}'
# Delete old snapshots - NOTE THIS DELETES ALL SNAPSHOTS THAT ARE NOT ASSOCIATED TO AN AMI!!!
ec2-describe-snapshots | sort -k 5 | awk '{print "Deleting-> " $2; system("ec2-delete-snapshot " $2)}'

    This is what you would use if you are using Access Keys:

    
#!/bin/bash
# Delete AMI's older than 7 days (must have the date in their name)
ec2-describe-images -O YOURACCESSKEY -W YOURSECRETACCESSKEY | grep `date -v -7d +%F` | awk '{print "Deregistering-> " $2; system("ec2-deregister -O YOURACCESSKEY -W YOURSECRETACCESSKEY " $2)}'
# Create AMI's from existing instances without rebooting them and putting the date next to the name
ec2-describe-instances -O YOURACCESSKEY -W YOURSECRETACCESSKEY | grep instance | awk '{print "Creating -> " $3; system("ec2-create-image --name " $5 "-$(date +%F) --no-reboot -O YOURACCESSKEY -W YOURSECRETACCESSKEY " $3)}'
# Delete old snapshots - NOTE THIS DELETES ALL SNAPSHOTS THAT ARE NOT ASSOCIATED TO AN AMI!!!
ec2-describe-snapshots -O YOURACCESSKEY -W YOURSECRETACCESSKEY | sort -k 5 | awk '{print "Deleting-> " $2; system("ec2-delete-snapshot -O YOURACCESSKEY -W YOURSECRETACCESSKEY " $2)}'

    *Your EC2 instance must be named, with NO spaces, for the AMI to be created with ec2-create-image using this script
    ** TO RUN SCRIPT FROM CRON, ADD ENVIRONMENTAL VARIABLES FROM STEP 4!

    9. Give your script execute privilege:

    
mac-mini:~ User$ chmod +x ~/Scripts/ec2-backup.sh

    10. Add the script to crontab to run at 1am (system time) everyday:

    
mac-mini:~ User$ crontab -e
#Backup script run at 1am 
0 1 * * * /Users/User/Scripts/ec2-backup.sh

    If you need any help, just leave me a comment and I will try to get you all sorted out.

    **UPDATE**
    With the update of the ec2 cli tools, there is now a new way to go about doing this using .json files. When time permits I will post how to make a cron ec2 backup job using the outputted .json file.




    Share
    9 comments , , , ,
     

    9 responses to “Simple Bash Automation for EC2 Instance Backups using Cron on a Mac Mini” RSS icon

    • Naresh Chidipothu September 15th, 2013 at 01:34

      Hi,

      I tried as you mention, but my cron job is failing with some error. it’s like “ec2-describe-images” and “ec2-describe-tags” not found. but thing is if run manually, it is working properly.. Could you please help me.

      Thanks,
      Naresh Ch

    • Hi Naresh,
      You have to make sure that the user you are running the CRON task under can access the tools. Usually they just don’t know where the tools are as the PATH is not set properly (you are setting it when you log into the terminal though hence why manual works, just not with crontab).

      One solution is to give the FULL path to your EC2 command in the script ex: /user/myuser/.ec2/bin/ec2-describe-images

      Alternatively, you can add PATH into your crontab at the top ex: PATH=/user/myuser/.ec2/bin/

    • Thanks for above script , can you describe following , will it terminate existing instances running more than a week ?

      # Delete AMI’s older than 7 days (must have the date in their name)

      ec2-describe-images -O YOURACCESSKEY -W YOURSECRETACCESSKEY | grep `date -v -7d +%F` | awk ‘{print “Deregistering-> ” $2; system(“ec2-deregister -O YOURACCESSKEY -W YOURSECRETACCESSKEY ” $2)}’

      How can i exclude dev.xxx instances for backup ?

    • Hi Mudassir,
      No it will not terminate anything, this just eliminates the SNAPSHOT and AMI you made, it does not terminate a running instance.

      If you want to exclude dev.*, I believe you can add in another grep -v. ie:

      ec2-describe-images … | grep `date -v -7d +%F` | grep -v dev.* | …

    • /opt/aws/bin/ec2-stop-instances: line 9: EC2_HOME: EC2_HOME is not set

      WHY! I’ve been struggling with this problem for a couple of days. Cron is not running the aws commands. Keep getting emails that EC2-HOME is not set for the cron shell.

    • Hey Sam, Sorry for the delay here. THe problem may be that the cron job is running as a different user than what you are logged in as. For example on Ubuntu, if you are logged in as user and your ec2 tools work while logged, but when cron tries to run it and it does not work you may have to adjust the settings on the root user. I believe it is also possible to set the EC2_HOME in the script file that you are trying to run with cron as a work around, so set your environment variables in there as well!

    • In my case it is giving error while executing

      date: invalid option — ‘v’
      Try ‘date –help’ for more information.

      I am using ubuntu

    • Rajan,
      Syntax in ubuntu is different than OS X. To subtract days you would have to use –date (-d). Example: date –date=’-7 day’ or date -d=’-7 day’

      You are also going to have issues with %F since that is also UNIX. For Ubuntu I believe if you want to format at say year-month-day you would have to do it like this:

      date –d=’-7 day’ +”%Y-%m-%d”

    • Oh my goodness! Incredible article dude! Many thanks, 


    Leave a reply

Categories

 

Computer Plumber | Computer Service Excellence