Unit Testing Shell Scripts:
Part Five

Previously in this series, we’ve discussed the value of unit testing shell scripts and explored how to roll our own test script, how to use the shunit2 library, how to use bash-spec for a behavioral-style approach, and how to use Pester to test-drive a Powershell script. Now we’ll look at two unit test frameworks for the server provisioning tools, Chef and Puppet.

Value and Limitations of Unit Testing Provisioning Scripts

In general, functional checking performed early in the delivery pipeline is faster and cheaper than checking performed later in the pipeline. The test cases can be of very small scope and can be isolated from external dependencies.

With that in mind, the main value in running unit-level checks of provisioning scripts is to catch trivial mistakes such as typos and overlooked configuration settings before the script is run against a live server, where it will be more tedious to track down the source of the error.

Unlike many other kinds of scripts, a provisioning script doesn’t take any real action when it is isolated from external dependencies; the whole purpose is to create an entity that will be an “external dependency” of other components in the environment.

For that reason, there’s a natural limit to the value we can obtain by unit-checking provisioning scripts. We can check that the script attempts to do the things we think we want it to do, based on how much we understood about how to properly configure the target system at the time we wrote the script.

So, the value and urgency to write comprehensive automated unit-level checks of provisioning scripts may be less than for scripts that perform hand-rolled logic. Depending on the tooling used, many provisioning scripts are little more than manifests of the packages and other software that should be present on the target instance. Pivotal Cloud Foundry even uses that very term to describe a config file that drives provisioning: manifest, with the filename manifest.yml.

The tooling (like Chef and Puppet, or any given cloud provider’s tooling) performs the actual work. Responsibility for testing those tools lies with the tools’ owners and operators, not with the end users (that’s us).

Possibly the greatest value in a set of executable tests or checks for a provisioning script is that they provide up-to-date and accurate documentation of how each instance in the environment is intended to be configured. Executable tests offer the best low-level technical documentation for any software solution, whether it’s a simple script or a large-scale application, because if they get out of sync with the code, the tests will fail and notify engineers that something is wrong.

Chef and ChefSpec

Chef uses cooking terms, like “recipe” and (of course) “chef,” so when you search for information online you’re likely to get a lot of results pertaining to actual recipes and chefs. As Chef is written in Ruby, you can eliminate some of the clutter from search results by specifying “ruby chef” rather than just “chef” in your search terms. To save you a little time, here are some key references about Chef and ChefSpec:

• Chef home page
• Chef Github project
• Learn Chef site (good tutorials)
• ChefSpec Github project
• ChefSpec section in the Chef docs
• Introductory article about ChefSpec by Aleksandar Diklic
• Ruby home page (Chef is written in Ruby)
• Rspec home page (ChefSpec is based on Rspec)

ChefSpec Example

Here’s an example of ChefSpec from one of my side projects. The project is a set of scripts for provisioning and configuring a lightweight software development environment based on Ubuntu Server, a Linux operating system distribution. I use Ubuntu Server as a base because it’s quicker than building from source and yet doesn’t have a heavyweight GUI environment or end-user applications pre-installed.

One of the things I need to add to the base to provision a development environment is X Windows support. The provisioning script uses Chef to install it, and ChefSpec to check the Chef recipe.

If you’re familiar with Rspec and Ruby, then ChefSpec and Chef will look pretty intuitive. Hopefully, they look reasonably understandable even to those who aren’t familiar with Ruby. It’s common for Rspec setups to have a ‘helper’ file that is automatically pulled into all the spec files. Here’s the spec_helper.rb file for this example:

require 'chefspec'
require 'chefspec/berkshelf'

A Chef recipe looks declarative, like a manifest, but the keywords are actually Ruby method names, and the recipe is executed sequentially from top to bottom; so it’s really imperative. That’s consistent with the cooking metaphor. If you performed the steps in a real recipe out of sequence, the food wouldn’t turn out well.

Here’s the Chef recipe to install X Windows on an Ubuntu Server instance, from the sample project. The filename is install_x.rb. It installs the X server, X client, the OpenBox window manager, and the lxterminal terminal emulator. The recipe also copies an OpenBox config file that contains the setting I prefer. Of course, we could factor out OpenBox and lxterminal to provide cleaner separation of concerns, but let’s save that for another day.

# Install X Windows client and server and openbox window manager.

package 'xauth'
package 'xorg'
package 'openbox'
package 'lxterminal'

bash 'copy openbox configuration files to user dev' do 
  code <<-EOF
    mkdir -p /home/dev/.config/openbox
    cp -r /root/bootstrap-ubuntu-server-16.04-dev-base/openbox/* /home/dev/.config/openbox/.
    chown -R dev /home/dev/.config/openbox
    chgrp -R dev /home/dev/.config/openbox
  EOF
end 

Here’s the ChefSpec unit test file for that recipe. The filename is install_x_spec.rb.

require 'spec_helper'

describe 'ubuntu_prep::install_x' do
  context 'Ubuntu 16.04' do
    let(:chef_run) do
      runner = ChefSpec::SoloRunner.new(platform: 'ubuntu', version: '16.04')
      runner.converge(described_recipe)
    end

    [ 'xauth', 'xorg', 'openbox', 'lxterminal' ].each do |package_name|
      it "installs package #{package_name}" do
        expect(chef_run).to install_package("#{package_name}")
      end
    end
  end
end

ChefSpec does not affect any actual target instances. When the spec is executed, it will verify that the recipe attempts to install the specified packages. This is only a unit-level check; it doesn’t guarantee the packages are actually installed properly on target instances.

This is conceptually analogous to using the “verify” feature of a mock to check that the code under test calls (or doesn’t call) a collaborator. It doesn’t confirm an actual result was produced.

Puppet and rspec-puppet

Puppet is another provisioning tool, also written in Ruby, and a unit testing framework is available for it, also based on Rspec. The principles, value, and limitations are the same as for Chef and ChefSpec.

There’s an excellent introductory article by Joseph Oaks on the Puppet site: Unit Testing Rspec-Puppet for Beginners. Rather than reiterate the contents of that article here, I’ll suggest you read it on the Puppet site.

Those of you familiar with rspec will find rspec-puppet even closer to plain old rspec than ChefSpec. It provides a readable and maintainable syntax for specifying the checks you want to perform.

Validating Server Configuration

Unit-level checks can catch simple errors in provisioning scripts and manifests, but they don’t check target instances to ensure they’ve been configured as expected. The subject is a little out of scope for this series, but bears mentioning briefly.

For Windows instances, you can use ServerValidator to check the configuration of servers after your provisioning scripts have run.

For Linux instances, you may have to write your own shell scripts or programs to validate server configurations. Fortunately, many server products support a “test” option that will verify they are set up to run in a reasonable way without actually doing anything to the target instance. It’s most often implemented as command-line option -t.

This article presents over 20 examples for checking the configuration of various web servers. You can run these commands from inside a verification script that you write in any shell language or scripting language.

Another approach is to write a script that checks whether specific packages are installed. Here’s an example from an old project of mine that I used to set up Java development instances for training. Take a look at the verify script in that project.

Version Control

We promised to mention version control in this series, at least. When we adopt development practices for system administration and operations, one of the implications is that our configuration files and scripts will be housed in a version control system, just as application source code and test scripts are kept under version control. We need to get used to managing our code in a robust way, rather than keeping scripts in individuals’ private directories, and hacking on them without tests and without the ability to revert to a known good version.

Conclusion

We’ve looked at several general-purpose unit test libraries for shell scripts as well as hand-rolled a test script without using a library. The conclusion is that there’s no reason we can’t get value from application development practices like unit testing and test-driven development for system administration and server provisioning scripts, as well.

System administrators and infrastructure engineers don’t always use shell languages. Ruby and Python are also used; each of those languages has more than one unit testing framework available for it. Perl is another language often used for scripting. We won’t be examining test frameworks for these languages in the present series, but there are a number of options for each language.