example master/slave on Ec2/S3

Great article from RightScale on an implementation of master/slave replication using Ec2/S3.

Not a lot of details, but a great high level explanation of what you want to accomplish.

Redundant MySQL set-up for Amazon EC2

What we’ve built is a mysql master/slave set-up with backup to Amazon S3. The set-up consists of one mysql master server which is the primary database server used by the application. We assume it runs on its own EC2 instance but it could probably share the instance with the application. We install LVM (linux volume manager) on the /mnt partition and place the database files there. We use LVM snapshots to back up the database to S3, this means that we get a consistent backup of the database files with only a sub-second hiccup to the database.

Advertisement

More EC2 Tools/Notes

Couple of random notes…

1) AWS Tools – This is a great set of command line tools for accessing S3 and EC2. I’ve only played with the S3 features, but they are great and make it a lot easier to implement some of my backup and restore to s3 features I am building on my Ec2 cloud/array

2) Ec2 First Run – Ok, this is totally obvious when you think about it, but it took me a couple days of scratching my head wondering why this wasn’t working before I had the “duh” moment. Mind you, I didn’t work on trying to solve this problem for 2 days, because I was working on other things… but it wasn’t obvious to me for a couple of calendar days. What the heck am I talking about!?!?! This:

• I wanted to build a custom AMI that would do a bunch of stuff on first run. Mostly create my lvm volumes, download a backup of my Database from S3, send me notification that it was up and running, stuff like that.
• The rc.local of the existing AMI has this nice little section of code labeled “first run” and it purports to do stuff on first run only, while the rest of rc.local would run on a reboot of the instance as well. Simple right? So I code up my first run shell scripts and get them all happy, drop them in the if branch of the code… build my AMI… launch a new instance… and (silence) nothing happens…. huh? Why’s that. Let’s look at the code:

  #!/bin/sh
  #
  # This script will be executed *after* all the other init scripts.
  # You can put your own initialization stuff in here if you don't
  # want to do the full Sys V style init stuff.
  
  # Stuff we want to do once at launch and never again:
  if [ -f "/root/firstrun" ]; then
  
      # do my stuff here --
      # create our logical volumes
      create-lvms.sh
  
      # restore from my latest backups
      restore-database-from-s3.sh
      restore-blogdata-from-s3.sh
  
      # send our admin team a notification
      firstrun-notify.sh
  
      ...
      # do some more stuff here -- from the orginal AMI
  
      rm -f /root/firstrun
  fi
  

• Well, it should have been obvious, but this code only runs if a file “/root/firstrun” exists, and at the end of the code, the code deletes that file. Well, since I made this AMI from a running instance, the first run had already run, and therefore “/root/firstrun” didn’t exist… DUH!
• So, I touched /root/firstrun and rebuilt my AMI, and now first run actually works. I need to remember to do this before I rebuild my AMI again.

More Notes on EC2 and LVM

Following up on my notes about running WordPressMU on EC2, I am still working on a clean implementation for setting up the disks on the EC2 m1.large instance the way I want. Once, I’ve made things work the way I want then I will go back and finish my other notes on running WordPressMU.

This is a collection of notes on what I think I want the disks to look like on first boot and reboot…

m1.large instances come with 850GB of disk space configured as:

• 850 GB instance storage (2 x 420 GB plus 10 GB root partition)
• /dev/sdb (/mnt on large and extra large instance types)
• /dev/sdc (on large and extra large instance types, not mounted)

I like the idea of creating a single large volume group on both physical disks, let’s call it “vgall” and then creating all of our logical volumes on that volume group:

• two database volumes (“lv_master”/”lv_slave”)
• one volume for back up (“lv_backup”)
• one volume for blog data (“lv_blogdata”)
• one swap volume (“lv_swap”)

Ok, so, we can’t create these logical volumes in the AMI setup, but we can create a script to run in rc.local that will set these up for us, and we can add these volumes to fstab to survive reboot.

So, our setupscript…

###  BEGIN OF SCRIPT ####!/bin/sh
#
# Script to make EC2 /mnt into a LVM volume

umount /mnt
pvcreate /dev/sdb /dev/sdc
vgcreate vg /dev/sdb /dev/sdc

lvcreate -L150G -n lv_master vg
mkfs -t ext3 /dev/vg/lv_master
mkdir -p /data/mysql_master
mount /dev/vg/lv_master /data/mysql_master

lvcreate -L150G -n lv_slave vg
mkfs -t ext3 /dev/vg/lv_slave
mkdir -p /data/mysql_slave
mount /dev/vg/lv_slave /data/mysql_slave

lvcreate -L150G -n lv_backup vg
mkfs -t ext3 /dev/vg/lv_backup
mkdir -p /data/mysql_backup
mount /dev/vg/lv_backup /data/mysql_backup

lvcreate -L150G -n lv_blogdata vg
mkfs -t ext3 /dev/vg/lv_blogdata
mkdir -p /data/blog-data
mount /dev/vg/lv_blogdata /data/blog-data

lvcreate -L140G -n lv_swap vg
mkswap /dev/vg/lv_swap
swapon /dev/vg/lv_swap

lvdisplay

mkdir -p /data/mysql_master/data
mkdir -p /data/mysql_slave/data

chown -R mysql:mysql /data/mysql_master
chown -R mysql:mysql /data/mysql_slave
chown -R mysql:mysql /data/mysql_backup
chown -R apache:apache /data/blog-data

### END OF SCRIPT ###

Our fstab:

# /etc/fstab
# modified to include lvm and swap

# original items
/dev/sda / ext3 defaults 1 1
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0

# our LVM volumes and such
/dev/mapper/vg-lv_master /data/mysql_master ext3 defaults 0 0
/dev/mapper/vg-lv_slave /data/mysql_slave ext3 defaults 0 0
/dev/mapper/vg-lv_backup /data/mysql_backup ext3 defaults 0 0
/dev/mapper/vg-lv_blogdata /data/www ext3 defaults 0 0
/dev/mapper/vg-lv_swap / swap defaults 0 0

Some supporting notes:

• LVM can’t be set up and included in an AMI, so you need to write a script to set up the Volume groups and the logical volumes and run it in rc.local (Notes here)

• Notes on fstab – Take note of the comment on running swap on an LVM managed disk.

Experiments with Running WordPressMU on EC2

Now that Amazon offers fixed IP addresses for their EC2 service, I am starting the process of researching what it would realistically take to implement WordPressMU on EC2.

I’m going to capture some notes here about the process. This post is a raw unfiltered list of random thoughts.

I am not writing this per se for anyone else, but since I’ve found these items out there and they’re useful to me, they may be useful to someone else. I may assume you already are very familiar with WordPressMU and/or MySQL, and certainly that you got admin skillz if you’re reading this.

Meta Point: You have no dedicated disk drive!

So the big issue with Ec2, and arguable grid computing vs dedicated servers is that you’re server instances don’t have a physical hard drive associated with them. Sure, they have disk space when they’re running, but as soon as they shut down that disk image is gone FOREVER… Obviously, if you plan to run a database on one of these machines, you have to think about this carefully.

It’s not like if you accidentally reboot your dedicated server at least you know your DB probably shut down safely, and will probably wake up working just fine. That disk is dedicated to you. But not so with EC2… that disk doesn’t exist anywhere. So you need to really think about how to make that disk image persistent.

But wait a second? Is that really an issue?

Well, if you’re lazy, or even if you’re not lazy and you’re counting on regular backups and the idea that your system could lose some data and you’d be ok… then maybe you haven’t built a system that withstand a complete drive failure. You figure, hey, I’ve got my dedicated box, I do regular backups, I copy those backups off of the box… Maybe I’m already using S3 to store those backups… no big deal. I could rebuild my box if I needed to.

Well, if that’s your attitude, then you would be flying pretty much by the seat of your pants on a pure cloud system like Ec2.

But, if you already have a redundancy and backup strategy that can handle a complete disk failure (as in there is nothing left on the disk) then you’re in perfect shape for running in the clouds.

So… here are some notes that are mostly EC2, MySQL, Linux specific about implementing various pieces of this puzzle.

Some great articles out there

General EC2 Articles

• The Official Amazon Ec2 Getting started guide -This is a pretty basic article but it explains everything you really need to know about setting up a basic EC2 instance.
• The Official Amazon EC2 Documentation (v2008-02-01) – This contains the actual documentation for all of the command line utilities, etc, that you need to get rolling. I’m the kind of guy that likes to know all the parameters available to me, so if I was walking through a how to guide that’s saying what to type in where, I still want to know all the options and why they work the way they do.

Mysql Database “strategy” articles:

• CodeWord’s Article on Running MySQL for large datasets on EC2 – This article is more than a year old, so it describes limitations in Ec2 that have been fixed, but the concepts of running MySQL in a double paired master/slave mode is a clever solution and totally appropriate for the project I’m working on.
• MySql documentation on mysqld_multi – This is the tool you’ll want to use when you run multiple mysqld instances on a single machine.
• Setting up Master-slave
• Setting up Master-Master

Articles about LVM backups:

• DBA Dojo’s Article on Setting up EC2 with LVM snapshots – Also totally appropriate, includes some handy scripts which demonstrate things like setting up your logical volumes, etc.
• PointyHair Article on Snapshot backups – Here’s another article about snapshots. It’s interesting as it shows how to do this from mysql, using mysqls ability to call shell commands in between SQL commands. This article also describes the (should be obvious) aspect of actually doing the real backup on the snapshot results…. so for example this is where you might copy to content to S3 using one of the below products.
• Another article that talks about LVM, MySQL Replication, and backups.
• Mike Kruckenberg’s post on the topic of LVM snapshot backups.

Articles related to file systems useful as “backing stores”:

• Great Thread Discussing different types of “backing stores” using S3 to make your a persistent image of your Ec2 data.
• ElasticDrive – Tool mentioned in the above thread… not exactly free, but looks like it has some useful features at scale.
• JungleDisk – Tool mentioned in above threads… practically free, solves a slightly different problem then ElasticDrive, but certainly would work nicely in the model described by CodeWord.
• PerstistFS – This is a FUSE based file system designed for S3/Ec2

Setting up my Image

I wanted to utilize the EC2 m1.large type for my database so that I would have plenty of room for DB performance. And I wanted to use Fedora Core (6 or 8).

So I started with the amazon public AMI for fedora core 6, which pretty much doesn’t include anything but the OS. I then installed: httpd, mysql, php, and subversion (since we like to update our code from our svn server). [hint: yum install httpd mysql mysql-devel mysql-server php subversion]

From this I made my own image that was fedora core, plus the basic LAMP components. This means following the standard directions building a bundle uploading it to S3, registering it, etc. Now I can start up new instances without having to reinstall all that stuff.

Installing WordPress MU: The code

Well, this is actually pretty easy, I just checked out our version of wpmu from our svn server. Our version includes all of our custom plugins and themes we’ve built.

Installing WordPress MU: The database

More to come.

Random Issues:

• For some reason I had some problems with the default amazon image for fedora core 6 when using an assigned IP attempting to do yum. I kept getting manifest checksum errors. Not sure if it’s a F6 issue or an assigned IP issue. I was able to yum a bunch of stuff with both FC6 and FC8 when not assigned a fixed IP. I had no problem when I recreated the instance (without associating it with the IP) and was able to yum and ultimately create an image I wanted. But it was a pain. Since then I’ve also been able to yum from an instance that is assigned to an IP… so really I have no idea what was going on here.