Automated server provisioning for rapid disaster recovery

Does this sound familar? In this situation, how would you and your team respond?

Often the most efficient (and sometimes the only) option is to set up a new server. Many teams will have written instructions for manually installing and configuring a replacement server (time consuming and difficult to maintain). Other teams won’t have anything (time consuming and embarassing). Most teams won’t have actually practised such a scenario.

Many server provisioning tools have a relatively steep learning curve and are overkill for small-medium enterprises. SaltStack (saltstack.com) is a product that is perfectly suited for SMEs and will gracefully scale to cover much more complex deployments.

You create Salt Formulas (plain-text files) to describe different components of your infrastructure: SSH keys, config files, software packages, databases, firewall rules, user accounts and much more. As an example, a simple Formula to install and configure PHP might read something like:

install_and_configure_php:
    pkg.installed:
      - pkgs:
        - php5-cli
    file.replace:
      - name: /etc/php5/cli/php.ini
      - pattern: short_open_tag = Off
      - repl: short_open_tag = On
      - count: 1

You choose which Salt Formulas are applied to which servers by using the Salt Top file. The Top file matches Formulas to servers based on the server hostname. A simple Top file might read something like:

base: '*':
  - ssh-keys
  - user-accounts
  - firewall
'web*': 
  - git
  - nginx
  - php
  - firewall-allow-www
'sql*':
  - mysql
'vcs*':
  - gitlab
  - firewall-allow-www

Provisioning a new server can literally be done in under a minute: you set a hostname, install the Salt Minion (compatible with Mac OS X, Windows and most flavours of Linux) and point the Minion at the Salt Master. On the Salt Master you accept the connection from the new Minion and then run a command to apply the required Formulas to the new server.

It’s easy to build, test and perfect your Salt Formulas and Top file using virtual machines. As an added bonus your Formulas and Top file are human-readable, making it easy to translate into written instructions suitable for manual implementation.

Documenting a recovery solution based on a provisioning tool like SaltStack is trivial. This makes the documentation really  easy to follow and unlikely to go out of date (don’t you hate stale docs?). As an added bonus, when it comes time to rebuild everything from scratch you can literally sit back and watch your computers do all the work.

William Pearse is the Software Lead at McKay. William has a BE Hons. and has worked at McKay since 2011. The software team works alongside the engineering team and covers many disciplines.