Key Development Techniques for Building Microservice-Based Solutions

Many contemporary business solutions take the form of a set of microservices that interact with one another in somewhat unpredictable ways in a dynamically-managed elastic cloud (or cloud-like) environment. This is a rather different architectural pattern than those from previous years, such as model-view-controller client/server solutions or batch extract-sort-edit-update solutions. Does this architectural pattern have characteristics that should lead us to reconsider our design and development methods?

To answer that question, let’s consider what we know about the reliability of such systems, and the factors that tend to assure high reliability. Then we may be able to identify techniques or methods that help us build reliable solutions.

Simple Testing in the Small

From Yuan et al:

• …simple testing can prevent most critical failures in distributed data-intensive systems.
• While professionals usually test that their…code works when things are going well, they rarely test that it does the right thing when something goes wrong. Adding just a few such tests during development would prevent a lot of pain downstream.

This finding is consistent with the widely-held idea in software circles that a “testing mindset” is one of the distinguishing characteristics of a software developer as opposed to a programmer or coder. A programmer tends to “test” their code to make sure it will work under ideal conditions. They tend to assume it’s “someone else’s problem” to create those ideal conditions. A tester tends to look for unplanned behaviors or behaviors outside the design parameters of the code. A software developer does both (among other things).

The Interaction of the Parts of a System

From Russell Ackoff (paraphrased): The behavior of a system depends on the interaction of its parts, not on the characteristics of each part in isolation.

This observation is consistent with a point of view about unit testing that is held by many proficient and successful software developers: To be meaningful and useful, testing must exercise the behavior of components interacting with one another. Compared to that, testing of individual code “units” in isolation offers little value. For a good presentation of this perspective, see “Why Most Unit Testing Is Waste,” by James Coplein.

Therefore, it seems worthwhile to consider:

• Thorough testing of non-happy-path scenarios “in the small” is a strong hedge against runtime errors “in the large” (Data science and software engineering). This derives from the finding that simple testing of the behavior of small components under error conditions helps avoid errors in complex systems built from those components.
• Rigorous control of interactions between services is a strong hedge against apparently-random emergent behavior in complex systems such as microservice fabrics (Russell Ackoff lecture on systems thinking). This derives from the observation that the interaction of the parts of a systen have significant effects on the overall behavior of that system.

TDD as a Way to Implement Yuan’s Findings

Bill Caputo has made interesting observations about the purpose or goal of “unit testing” as it applies to test-driven development (TDD). He concludes TDD is a technique for developing a specification incrementally, with tight feedback loops involving developers and stakeholders. This contrasts with the conventional view among practitioners that TDD is a software design technique, as well as with the common misunderstanding on the part of non-practitioners (and novice practitioners) that TDD is a testing technique.

Robert Martin’s Transformation Priority Premise is a result of his many years of experience in applying TDD. It proposes a sequence in which we ought to expand the functionality of code under development, guided by microtests. The approach offers a way for us to emerge a low-level design for a unit of code that performs the required functionality in the simplest way and with the least risk of overengineering. It is a way of using TDD as a design technique.

Based on Caputo’s observation, we know we can use TDD as a way to develop a clear specification for what a unit of code should do. Based on Martin’s work, we know we can use TDD as a way to guide an emergent design for a unit of code. The implication is that TDD may be a practical mechanism to ensure each small unit of code does what it is meant to do and has high reliability under various error conditions. In other words, TDD is a way to apply the finding of Yuan et al that thorough small-scale testing of each unit of code contributes significantly to reliability at scale when the unit is included in a complex solution. To produce this form of value through TDD, developers must cultivate a testing mindset to complement their programming mindset.

Design by Contract as a Way to Support Systems Thinking in Solution Design

Design by Contract (DbC) is an approach to development that focuses on defining the interactions between components and enforcing the rules of those interactions (the contract) at runtime. The Eiffel programming language was created to support this approach directly through language constructs, and DbC is described on the website of the Eiffel language.

Each interaction between software components involves a client that needs a benefit from a supplier. In order to request the benefit from the supplier, each client has certain obligations. The obligations are called preconditions. Clients are responsible for ensuring the preconditions are true; suppliers may assume the preconditions are true. Suppliers are responsible for guaranteeing postconditions are true. Postconditions are things that must be true after the benefit has been returned to the client. Clients then must assume the postconditions are true.

The Eiffel language is explicitly designed to support DbC. However, DbC can be used with any programming language provided developers follow certain conventions.

DbC was originally conceived as a way to assure reliable object-oriented programs. Note the similarities between the interactions between client and supplier objects in an OO program, and the interactions between clients and services in a microservices fabric. We can see that a DbC approach to microservices design is a practical way to support systems thinking in the design of microservices.

DbC is not a “testing” technique and cannot be mistaken for one because the guarantees of preconditions, postconditions, and invariants (not described here) happen at runtime, not at build time.

Supporting Techniques

In an entertaining and enlightening talk, Greg Wilson explores What we know about software development and why we believe it’s true. Wilson mentions several practices and techniques we’ve come to depend on to gain confidence in the correctness of our code.

It turns out the one practice that actually helps is an old one: Code review. It turns out, as well, the one metric that actually correlates with code quality is an old one: Number of lines of source code.

Notwithstanding our fondness for (and habits around) other techniques and other metrics, these two factors seem to have the greatest correlation with reliable systems.

So it may be interesting to ask:

• What is a practical and efficient way to gain the benefits of code review?
• What is a practical and efficient way to ensure our code modules are of small size?

First, here’s some more information about effective code reviews:

• It doesn’t help to have multiple reviewers. The 2nd through nth reviewer add nothing beyond what the first reviewer finds (in the studies cited).
• A reviewer grows mentally tired after about one hour. When the amount of code to be reviewed is more than can be inspected in depth in one hour or less, the reviewer will overlook errors.

We tend to favor Lean Thinking in crafting our software delivery processes. The traditional way to implement code reviews is to assign a technical lead or team lead to review each team member’s code. This creates a bottleneck in the delivery process as developers wait for a senior team member to become available to conduct a review. These pauses or halts in the delivery pipeline amount to waste, according to Lean Thinking.

Is there a way to provide for code review without introducing a pause or halt in the delivery pipeline? Fortunately, there is a simple way: Pair programming. One of the effects of pair programming is continuous code review. It’s a low-cost technique that has been shown to reduce defects by as much as 86% with very little additional overhead as compared with solo programming (in the range of 0% to 15% additional time, according to the seminal study of pair programmg by Alistair Cockburn and Laurie Williams at the University of Utah).

You might protest that every team member may not be skilled enough to review code. It may be the case that only the technical lead has that level of skill. Fortunately, it turns out that everyone on a technical team can learn the skill to review code, and it doesn’t take very long.

Is there a way to ensure code units don’t grow too large to be reviewable in less than an hour? Fortunately, pair programming offers a mechanism for this, as well. When a code unit seems to be growing beyond a “reasonable” size, a colleague is on hand to make that observation and to suggest appropriate refactoring of the code. Pair programming tends to reduce the tendency of developers to carry on with their design ideas without pausing to assess what they are doing.

Conclusion

When we want to assure high confidence in the reliability of a complex solution that comprises many microservices that are dynamically instantiated and destroyed by an elastic runtime environment, it seems reasonable to apply a short list of key techniques:

• The cultivation of both a testing mindset and a programming mindset
• Test-driven development of small building blocks of code
• Design by Contract of interfaces between software components
• Pair programming to provide continuous code review and “sanity checks” during development