Sporadically delivered thoughts on Continuous Delivery

Ready for Disaster

| Comments

There are a lot of things you can do to make sure that when disaster strikes, you can get back online. Even in environments where you don’t have automatic failover, you can take some basic steps so that when you get the alert or the phone call, you can bring things back online.

Let’s say you have a single server running a web application with a local database. However, you need to have a second server available. Maybe it’s doing something else normally, maybe it’s in a less than ideal location, like in your office at the end of a slower Net connection, but as long as you can fire up your application, repoint DNS, and be online, it’ll do in a pinch.

First, make sure you have the base server software ready, so your web, application, and database software are installed.

Second, make sure you have a copy of your application code and configuration files handy. I always like to have these in source control, on a server other than my live one, so in the worst case I can pull them down to my emergency location.

Third, you need your live data, that is, your database contents. Take frequent dumps of the data and have them handy, again away from your live server. Do this outside your system backups, use your database tools such as mysqlbackup to dump a file, then zip it and ship it somewhere else. How frequently you do this depends on how often the data changes, and how important it is to have fresh data. In the most extreme case, you might have the database continually dumping a log to a shared file store, where the backup server is reading it in.

A sticking point may be the DNS. You can change the DNS, but users will have the old DNS information cached. How long it takes for them to get the new IP address depends on the TTL you have set in your DNS configuration, and changing this to a lower value after the crash ain’t gonna help. A two hour TTL is probably a good setting.

Of course, better yet is if you have multiple servers behind a firewall and/or load balancer, so you don’t need to change your DNS at all, just reconfigure and go. But if you’re running a budget setup, these are simple steps to follow to make disasters a little less stressful.