Continuous Unit Checking:
Part One

• Part 1 (this): Command-line utilities and wrapper scripts
• Part 2: Guard (with Ruby) and SBT (with Scala)
• Part 3: Infinitest (with Java and IntelliJ IDEA)

Back in the day, we used to run compile, link, and execute steps separately and manually (after walking 15 miles to and from school in a snowstorm, uphill both ways). As integrated development environments (IDEs) matured, they wrapped those steps so that we needn’t do them manually over and over again. That was a great step forward, as we could maintain our focus on solving the problem at hand without being distracted by mechanical details of building our solution.

As test-driven development (TDD) and microtesting became mainstream practices, we started to feel those old distractions again. As we made small, incremental changes to our code and our microtests, we had to remember to click the “run tests” button or press the key sequence we had assigned to “run tests”.

A common problem TDDers run into is that we forget to run our unit checks after every change because our minds are occupied with solving the problem at hand. I’ve seen experienced practitioners do that even while giving demonstrations of TDD.

That’s as it should be. Our minds should be focused on the solution and not on the mechanical details of building the code and running automated checks.

Continuous Integration tools and practices give us that sort of automation at higher levels of automated checking. Is there a practical way to bake this operation into our tooling at the unit or micro level, as we did with compiling and linking?

Fortunately, there are several ways. They range from simple and crude to complicated and smart.

Let’s sneak up on the topic, starting with the most primitive approach and working our way forward.

In this installment, we’ll roll our own continuous testing solution using command-line utilities, and we’ll look at simple wrapper scripts.

Simple App for Initial Examples

We need something to run for the first couple of examples. Here’s what I used: A python script that increments an integer, and another one that runs a unittest example against it. Both files are in the same directory, and there’s no project setup or explicitly-defined namespaces.

Here’s the “application”. It’s increment.py.

'''
Groundbreaking application that will change the world
'''
 
def increment(a):
    return a + 1

And here’s the “test suite”. It’s test_increment.py.

#!/usr/local/bin/python3
import unittest
from increment import increment
 
class TestIncrement(unittest.TestCase):
 
    def test_it_increments(self):
        self.assertEqual( increment(1), 2 )

if __name__ == '__main__':
    unittest.main()

Command-Line Loop

For a quick-and-dirty continuous testing setup, you can write an infinite loop on the command line (not in a script) that runs your build/test commands on a timer. Here’s an example using bash:

while :; do ./test_increment.py; sleep 5; done

Those commands create an infinite loop (while :) in which the script ./test_increment.py is executed every 5 seconds. When you’re ready to stop you can interrupt the loop with Ctrl+c.

Why bash? It’s the default shell on most Linux systems as well as on Mac OSX (a Unix system), and it’s a common option on all Unix and Linux systems even if it isn’t the default. It’s included in the Windows Subsystem for Linux, and you can use it through Cygwin on Windows, if you prefer. So, it’s readily available on all mainstream systems a developer might use. But this is only one example; you can do something similar with just about any shell language.

Here’s a trivial example to give you a sense of what the work flow could look like.

Although a crude approach, there’s an advantage over pre-built continuous testing tools in that you can execute literally any code you want within the loop. That’s a special case of a general rule – if you’re willing to build something from scratch, you can make it do whatever you want, limited only by the amount of sweat you’re willing to invest. There are a couple of significant limitations, however.

First, you probably don’t want to run your tests unconditionally every n seconds. You may not have made any changes in that time; after all, most of software development work comprises thinking, not typing. Conversely, when you’re in a coding frenzy you may need to check your results more frequently than every n seconds.

Second, if you want to control the selection of test cases using any sort of robust heuristics based on how the modified code is related to other code in the code base, you have to build an application to do it. That’s a key value the more-sophisticated continuous testing tools offer. Working on a bare command line, you’re a long way from that sort of functionality.

Watch Directories for Modifications

A slightly better approach is to watch directories for changes and kick off a build/test script when something changes. Here’s an example I tried on Mac OSX using the utility, fswatch. First, install fswatch using homebrew:

brew update
brew install fswatch

Then, start fswatch on a command line and specify the script(s) you want to run every time there is a change to files in the directories you’re watching. This example runs a unit check script whenever any file changes in any way:

fswatch -0 . | xargs -0 -n1 -I{} ./test_increment.py

This tool allows for some control over the kinds of changes it should monitor. Like the first method, you can execute any script(s) at all, and you can include your own logic to decide what to do depending on the type of modification and which files were affected. See the fswatch documentation for details.

Here’s a trivial example to give you a sense of what the work flow could look like.

Fswatch is not the only tool of its kind. If it isn’t available for your platform you can use another. For example, inotifywait is available for Linux platforms.

The inotify-tools wiki has instructions for installing the package on all Linux distros as well as FreeBSD Unix. Slackware Linux includes inotify-tools by default. You can also build from source; procedure is documented on the wiki.

Here’s the same trivial setup as shown above for fswatch, but using inotifywait in an infinite loop with bash:

while :; do inotifywait -e modify,create,delete -r . && ./test_increment.py; done

But I’m Not Using Linux/Unix!

What about Windows? You can use the Linux subsystem or Cywin, but if you’d rather stick with native Windows facilities there are ways to accomplish the same goal. See this StackOverflow answer to get started. It talks about writing a FileSystemWatcher class in .NET to generate events when changes are detected. A little bit of sweat equity is required.

What about zOS? If you work with IBM mainframe technologies, you have access to many of the same tools on zOS as you have on Linux and Unix systems through Unix System Services (USS). Like most operating systems these days, zOS exposes a Unix-like shell that presents a command line and supports the usual Unix utilities. The default shell is tcsh, but you can get a bash interpreter from the collection of Ported Tools.

What about HP NonStop (formerly Tandem)? HP NonStop has a facility called Open System Services (OSS) that exposes a POSIX shell and provides the usual Unix commands. It supports korn, but not bash. You can write the same sort of command-line loop in korn as you can in bash. Most of the documentation for this system is published as PDF files. A good place to start is the OSS Shell and Utilities Reference Manual.

Wrapper Scripts

People who like the work flow of continuous testing at the unit or microtest level have written numerous scripts to wrap command-line functionality and make it easier to live with the automation. The simplest of these feel very much the same as using plain command-line tools.

One example by John Jacobsen, originally written to support Python, is called conttest. The project repo is on Github, and you can install it with pip. The procedure for installing pip varies by platform. Here’s how I installed it on a MacBook Pro:

curl https://bootstrap.pypa.io/get-pip.py > get-pip.py
sudo python get-pip.py

My understanding is that easy_install has been deprecated in favor of get-pip.py.

Then, installing conttest consisted of:

sudo pip install tornado
sudo pip install nose
sudo pip install conttest

Nose was required for the install to work, but conttest doesn’t require it. You can use any testing framework you wish – pytest, pyunit, unittest, and of course nose. Conttest worked fine for my simple example that uses unittest. It behaves the same as fswatch or inotifywait or a command-line loop.

conttest ./test_increment.py

Conttest offers options to exclude files by filename patterns, and doesn’t restrict you from running any code you wish to run when a change is detected in a file. You can use it for continuous testing of languages other than Python, too.

But you can do all the same things using standard command-line tools. What’s the advantage of a wrapper script?

There may be no great benefit to using a wrapper that only duplicates the behavior of a command-line loop, especially when our “application” is as simple as the Python example we’ve been exploring. I think you’ll find that as we climb the pile of tools we get more sophisticated functionality at the cost of more complicated configuration and limits on how much control we have.

In some cases the trade-off is well worth it; for instance, using Infinitest with Java inside an IDE provides good value, while while a similar degree of configuration effort might not pay off for other languages.

What’s Next?

In Part 2, we’ll climb higher on the tool pile and have a look at a couple of solutions that take the idea beyond simple shell scripting. The Ruby gem, guard, is a popular tool in that community, and sbt is a Scala-focused tool that can support Java (although few Java developers use it), and other languages through plugins. These are used with a command-line developer work flow, but they are more than wrapper scripts.

In Part 3, we’ll climb to the top of the pile and check out Infinitest, which works as an IDE plugin supporting JetBrains IntelliJ IDEA and Eclipse. It was originally created to support Java and Groovy. Like other tools in this category, it can be used with other languages, but most developers don’t use it that way. Infinitest is a practical choice for people who prefer an IDE-based developer work flow, and who work on Java projects.