Half Agile Isn’t Real Transformation
A Metaphorical Journey
As we guide clients through organizational Transformation, we apply our System of Transformation incrementally via Expeditions. An Expedition is a “journey” from the current state to a goal state, starting wherever the client organization may be in our Compass model, and striving to reach a milestone in the Transformation that we represent as a “Basecamp,” in keeping with a mountain-climbing or hiking metaphor.
Typically three to four months in duration, each Expedition involves a vertical slice of the organization. The slice will usually be aligned with a value stream, product, or product line whose desired Basecamp depends on the company’s orientation to its market as well as on the desired manner of internal operations designed to serve that market.
For example, a Basecamp 2 organization serves a market that can absorb change every few months, with a product line that is relatively predictable and stable, while a Basecamp 5 organization operates much like a lean startup, running experiments in the market and requiring the ability to turn changes around in a matter of days or, in some cases, even minutes.
Multiple Expeditions may be underway at the same time, each one involving a distinct vertical slice of business operations and the IT support for those operations. Some areas of the business may need to operate at a different Basecamp level than others. For instance, some facets of banking may serve their market quite well at a Basecamp 2 level, while others may need to operate at Basecamp 3 or 4 in order to prevent customers from seeking more responsive alternatives for their banking needs.
Tech Infrastructure != Value Stream
Many organizations divide their technology stack between front-end and back-end systems and place an API boundary between the two. The structure can be called by different names; it’s often called Service-Oriented Architecture (SOA). Applications that live in the front-end space access back-end resources by calling APIs, which in turn are serviced by components in the boundary layer.
This is a slightly broader definition of “front-end” that we normally hear from, say, web app developers. I’m including all IT resources “in front of” the API layer in the organization’s technical infrastructure. The usual arrangement has enterprise-scale, mission-critical, and often “legacy” resources in the back-end world, separated by the API layer from the more modern technologies in the front-end world.
The latter provides a nice-looking and usable interface to the “real” systems in the back-end, to provide read and write access to customers’ data via their smart devices, and to print materials to be mailed out, such as account summaries, late payment notices, and documentation required by regulatory bodies. With some exceptions, the front-end systems usually do not perform the “meat” of the computation or data storage that supports the company’s business operations.
Very often, a transaction traverses all the architectural layers and is partially processed by components that live in both the front-end and back-end worlds. If those worlds are not connected, then there’s a limit to the level of business agility the IT department can support.
Business agility is what organizations are looking for; agile software development may be one enabling factor in achieving it, but it isn’t the point of a transformation. Dividing your transformation initiative along the seams of the technical infrastructure will often be a suboptimal approach.
Half-Agile Transformations
Since the popularization of “agile” software development following the publication of the Agile Manifesto in 2001, thousands of companies have undertaken “agile transformations” or “agile adoptions” of one form or another. Many larger organizations have attempted multiple adoptions/transformations over the years, with mixed results.
If you know me, then you know I’m not a fan of the word “fail.” Yet, I might say a common “failure mode” in many cases is that agile methods, processes, techniques, tools, and thinking are applied only to the front-end environment, while the back-end is left as-is. Front-end applications based on technologies that easily lend themselves to agile practices (mobile, web, application server, lightweight databases, API client code, UIs and similar technologies hosting applications developed in languages like C#, Java, Python, Ruby, and PHP) can be modified and deployed smoothly with high frequency, while the back-end resources on which they depend are still delivered on the “old” cadence.
Most agile consultants and coaches are unfamiliar with back-end technologies, and they tend to treat them as a mysterious black box. That’s why you hear them refer to application servers and mid-tier lightweight database management systems as “back-end” resources, even though the true back end is farther back than that. HP NonStop? Never heard of it. IBM zOS? Hmm, I may have heard of that one. Can’t we just strangle it off? Yeah, that should be easy, right?
This may be perfectly fine in some cases, but in many cases, the result is not so fine: Any change that is meaningful and impactful to the business can be delivered only on the pre-agile cadence. The throughput of the system of delivery is limited to the capacity of its constraint. No matter how short the turnaround time for front-end changes, the corresponding back-end changes still require several months to deliver.
From the customer’s point of view, nothing has been delivered, even though the front-end changes are languishing in a staging environment, waiting for the back-end to catch up.
“We tried Agile, and it didn’t work.”
Value Streams May Cut Through Architectural Layers
The reason this happens is a product line or value stream (or whatever you choose to call it) may not be supported entirely within the front-end infrastructure. When determining which parts of the IT organization need to adopt “agile,” slicing the organization horizontally per technology stack may not align with the way business is done.
That’s why we identify vertical slices of the organization and determine which of them needs to function at a given Basecamp level in order to support business operations on a per-product or per-value stream basis, regardless of the technologies involved in each slice. It’s possible only some of your front-end teams really need to “go agile.” By the same token, it’s possible some of your back-end teams need to “go agile” along with them, to support business operations appropriately.
Is it easy? No. Technologies used for front-end development (in the broad sense) have tooling, training, and wide community support for things like Continuous Integration, Version Control, Test-Driven Development, incremental refactoring, Continuous Delivery, and production observability. Meanwhile, the back-end environment may have far fewer resources available off-the-shelf to support such things, and the resources that are available may have been designed to support relatively long release cycles (months) rather than minute-to-minute incremental commits to version control using trunk-based development, with multiple teams active on the same codebase. It can be challenging to get to that operational state.
Comprehensive transformation
Attempts to implement “agile” in a bottom-up fashion, beginning with team-level methods like Scrum or Extreme Programming, have achieved limited penetration organizationally, have focused exclusively on team-level software development practices without a larger business context, and have tended to fade or degrade with time. Purely top-down, management-driven “agile by mandate” initiatives have fallen apart for different reasons.
To avoid the “we tried, it didn’t” pattern, the agile transformation has to be driven by business concerns, and all the moving parts of the organization involved in those business concerns have to be transformed in concert, with due attention paid to organizational constraints and real-world challenges.
Which operations need to function at which Basecamp level (or equivalent by some other name)? Market analysis and business strategy know-how and institutionalized processes are required to answer that question.
How do we tease apart the specific platforms and applications that support each product line or value stream? Significant technical expertise in enterprise architecture, including knowledge of legacy technologies, is required to answer that question.
How can we introduce processes, tooling, techniques, and modes of thinking consistent with the target operational state of each organizational slice? An explicit system of transformation, as opposed to a system of delivery, is required to achieve this goal. It demands experience in training or teaching, as well as the nuances of coaching in the true sense, for all audiences and roles in the organization. All the better when that knowledge is based on patterns observed in many organizations that have attempted transformations.
You can embrace an organizational change model such as Kotter or ADKAR. That will give you a sound philosophical underpinning for organizational change, but no concrete advice on how to move the needle. Such models offer a starting point to develop a system of transformation, but they don’t actually provide one. In addition, they don’t describe any sort of target system of delivery to aim for, that might support your business objectives.
You can adopt a framework for “scaling agile,” such as SAFe, LeSS, or Disciplined Agile. That will give you a vision for the target system of delivery, but again no concrete advice on how to move the needle. Such frameworks define a system of delivery, but no system of transformation. Similarly, the Kanban Method will give you tools and metrics to help you discover opportunities for improvement in the organization but does not include any concrete advice on how to realize those improvements.
The various experiences of organizations that have tried different approaches to transformation point to this: We need a system of transformation that is distinct from the system of delivery. The system of transformation is specifically designed to guide change, rather than defining an end state operational model. At the same time, we need to have a well-defined operational model that supports our desired business objectives. Both pieces are necessary for success.