Some Thoughts on Agile Transformation in Big Companies
As usual… I’ve gotten a little sidetracked on my Agile 101, 102… series of posts. I’ve got the 201 post half finished, but haven’t been able to spend the time getting over the hump. That said, I want to divert just a little and talk about agile in general, explore some of what it is, and a little about where it’s going. I think we have created a bunch of confusion in the marketplace. I am under no illusions that this post will be the definitive explanation, but I spend a TON of time talking to people and level setting before I can have any kind of intelligent conversation on agile, so I want to explore some of the points that I think resonate.
Can We Define Agile?
First of all… back in my early, early days of community involvement at V1… I was really wresting with this untangling language around agile. I used to break the conversation into a few key areas: project management, product management, leadership thinking, and technical practices. Today, I find myself taking a slightly different angle. I tend to see agile discussed as a cultural phenomenon, sometimes as a set of practices, and sometimes as a way of structuring and governing how your company works. More often than not, agile is setup as a counterpoint to some of the most ridiculous examples of bad management we can come up with in an industry.
From my point of view, agile is all of these things collectively and none of these things individually. Agile is about having a rational system of work in place where people are engaged and can thrive, but one that produces the business outcomes that our stakeholders are counting on. When we get too myopically focused on one aspect over others, it is easy to start talking in platitudes and one-size-fits-all adoption approaches. Every organization I’ve ever worked with has been vastly different and, while certain patterns apply, solutions are always unique to the people that are impacted by the change.
The first line of the Agile Manifesto says that we are uncovering better ways of developing software by doing it and helping others do it. How we implement agile is supposed to change over time, and how it changes should be guided by the values and principles in the Manifesto. The challenge is that the values and principles are supposed to guide the practices and structures we choose to implement. The values and principles don’t live by themselves or in a vacuum. To successfully implement agile, we have to have a system of delivery and supporting practices which enable the values and principles to be lived out.
The Problem With Big Companies
I’ve been focused on what I call the ‘Big Company Problem’ since I left CheckFree more than 7 years ago. Back in the day, we were building complex systems of systems for the financial services industry. Products of products. Multiple overlapping and intersecting value chains. Heavy traditional governance and pockets of agile scattered throughout the organization. This was an environment with specialized business domain expertise, multiple diverse technology stacks, an organization that demanded 5 9’s reliability where downtime immediately impacted revenue and incurred penalties. How does one begin the process of agile adoption in any meaningful way in an environment like this?
Many of us suggest that ‘big and complicated’ is the problem and that we need to be ‘small and simple’. I agree… but, these companies ARE big and complicated so the question becomes HOW to help them become small and simple. Just saying BIG is bad and you should be SMALL doesn’t help. How do we get there? What is the path from A to B? Is adopting an agile value system enough? In the presence of the right value system, will the right delivery systems and practices emerge? In the presence of the right practices, can I impact the end-to-end system of delivery and make the cultural changes I need to make?
Culture First?
The problem with the culture first mindset is that there is practically no way to change enough hearts and minds to get everyone on board fast enough to produce results in the timeframes we are working against. Even if I could get everyone to adopt an agile mindset, we have real governance issues and audit requirements that often cannot be changed overnight. We have tightly coupled technology stacks, tightly coupled product requirements, imbalances in capacity and demand, and bottlenecks all over the place. Solving the problem is a matter of redesigning the system of delivery. Not everyone is an expert in designing systems, and even those that are don’t agree on the best path forward.
There are clearly some places where a culture first transformation can work. There are definitely some where it won’t. As an industry, we have to have an answer for when the scale and complexity are such that self-organization around the problem isn’t a viable approach. Again, the right company, the right business problem, the right underlying technology, and the right group of really smart people can get a ton of stuff done given the right degrees of freedom. There are some environments, though, where this is recipe for chaos. I personally think it is irresponsible to suggest that cultural transformation is going to fix this. We cannot always assume that if enough people ‘get it’ agile will begin to take hold.
Practices First?
Here is another one that I believe is driven by our long history of thinking of agile as a set of processes. Most top down organizations are accustomed to using process to pull together disparate working groups and coordinating activities across them to create larger deliverables or achieve broader organizational objectives. In many of these cases, the underlying organization, and maybe even the underlying organizational culture, doesn’t matter all that much. All we need to do is document a process that people can follow and, provided they actually follow it, we’ll get the outcomes that we desire. People assume that agile as a process will behave similarly.
Agile is different. You can’t take a set of functionally siloed teams, operating under heavy management control, traditional phase gate governance, fixed time, cost, and scope constraints, and tell them to do whatever process they want. You can’t tell them that agile is okay, but then hold them accountable to live in the same organization, under the same rules, and under the same constraints as a waterfall organization. The agile process in and of itself doesn’t make any rational sense outside the context of a team based organizational model, an agile governance framework, and a mindset that allows for some adaptation to the unknown. It doesn’t work.
Practices alone will never solve this problem. All we end up with is a bastardized form of agile where people go through the motions, but where agile doesn’t live up to the value that was promised. In order to do agile well, you have to have teams, you have to have backlog, and you have to have the ability to produce a working tested increment of product at the end of each iteration. This involves rethinking how you go about forming teams, how teams work together, and how value is coordinated. In the presence of teams and practices, I believe that the right thinking can emerge.
Structure First?
I’m a big believer that agile culture and agile practices are essential for long term sustainable organizational agility. That said, I don’t think you can train your way into it and I don’t think you can believe your way into it. I think you have to start with a team based organizational design where the flow of value is governed across teams using an adaptive governance model, based on lean/kanban principles, where WIP is limited, capacity and demand are balanced, bottlenecks are identified and dealt with, and management is invested in improving the overall system of delivery. Only within this kind of organization can an agile culture emerge and agile practices thrive.
I also don’t think that people intuitively understand these kinds of systems and I don’t think that most organizations will self-organize their way into them. Most people tend to think about optimizing what they can control and getting better only at what they are responsible for delivering. People are often self-interested and will consciously or sub-consciously advocate for system designs that are in their best interests. In the long run… forming teams, decoupling dependencies, reducing complexity, and creating an ecosystem where small, independent teams can self organize within their boundaries is the only model I believe will work.
So, I agree that we need to be small and simple, but getting to small and simple… much of the time… is a process of forming teams around business problems and technology, progressively decoupling them from each other, reducing dependencies, and dealing with bottlenecks. There are real issues facing large enterprises, there is a ton of momentum around the old way of doing things, real organizational and technological constraints, and just telling people to self-organize around a new system of delivery… one that they may or may not collectively understand… is often a recipe for disaster.
Structure then Practices then Culture
We do though have a bit of a chicken and the egg problem here. How do I get permission to start an agile transformation if I don’t have a leader with the right mindset to give it a try? I’d suggest that you at least have to have a leader that recognizes there is a problem and is open to alternative ways of solving it. I’d suggest you begin by focusing on your business goals, articulating a strategy to create a team based organizational model, and model based on iterative and incremental delivery principles, one that uses agile and lean methodologies for delivery and governance, but that operates within the operational and cultural constraints of the existing organization and it’s policies.
You teach this organization agile technical practices and management practices at the team level and adaptive lean governance practices at the program and portfolio level. You teach the organization how to deliver with reliability and predictability within that framework as you build trust in the new way of working. As we learn the capability of the new model and build trust with the organization, we create the opportunity to influence hearts and minds and create the environment where people feel safe to take chances and absorb risk with new ways of thinking that challenge their old ways of thinking.
The Role of Leadership
These kinds of changes have to be led with top down intent, but with a bottom up implementation. People have to be engaged and do ultimately have to be bought in. That said, buy-in will only come when people realize that the new system is working and the new system rewards the new behaviors. Once we have the rational team based system of delivery, new practices that support operating the new model, and a new mindset that enables us to unleash the power of the new system of work… we can start really improving as an organization and start tapping into the promise we’ve been reading about with agile all these years.
IMO… it’s not so important to talk about agile as structure, practices, or culture… it really is all three. It doesn’t matter if we are talking about agile as a set of management practices, an approach to product development, a leadership framework, or an approach to software craftsmanship. It is all of those things as well. But when it’s all those things living in an integrated system of delivery, where all the parts work together, when they are no longer in competition, where we really start to see the value. Things have to work as an integrated whole. It’s the working together as an integrated whole that is going to make this take off. Not just some team level agile practices or pockets of agile mindset in a vacuum.
Maybe this is all obvious, but there is a lot of debate about where to start with your agile transformation and what’s important to focus on first. If not debate, then different messages and approach in the marketplace around what works. I think that many folks work in broken organizations and can’t see a path forward… because without the support and direction of your senior leaders, nothing I’m talking about is even possible. As an industry though, once we get the messaging right, you are going to see more and more agile adoptions led from the C-suite, not in silly Dilbertesque caricatures, with informed intent around how to build organizations with a systems based perspective.
In most large organizations, bottom up is not an option without leadership acting from the top.
Comments (14)
John Coleman
Here here. I’ll re-comment when I write a post on Agile as a change, that is the combination of Agile theory and change theory, and perhaps how it could be applied in certain contexts. Little hint – it will cover push, pull, behaviours and playbooks. And it will touch on your post above. Essentially it might be supplementary material for anyone who is interested. Other people have written on scaling models and I will touch on that as well.
Andrew Binstead
Maybe the solution is actually these companies should have problems. It strikes me the reason they don’t change is because they don’t have to. They might be able to see they have problems but they work well enough to keep going. Until they feel the pain of loosing their market or not being competitive. Do they really feel the need to change? Then perhaps it will be too late to change.
A large company not being able to adapt quickly opens up the space for smaller more agile companies to come in and fill those gaps. Then the large organisation will shrink on its own and be force to accept more agile practices.
I totally agree with you, that change needs to come from the very top and they have to be committed to that change. They have to prove to the organisation the change is vital to the companies on going survival, and earn the trust of everyone to show them they can do it properly.
Hey Andrew… I’m not in the business of letting companies fail, but I hear you. I often ask practitioners if their CIO thinks their company has a problem. If the answer is ‘no’ there isn’t much anyone can do to help. You have to see the problem before you’ll invest in fixing it. Sometimes companies make money in spite of themselves so there isn’t any sense of urgency around investing to make things better. Thanks for your comment!
Mike Cottmeyer
Hey John,
Thanks for the comment. I’ll look forward to your addendum. Please leave us a link
Hala
I really enjoyed this article, Mike. You articulate the issue of the realities faced by companies and teams so well, and I love your approach and mindset. Thanks for sharing.
Marc Danziger (@marcdanziger)
My thinking is very much aligned w/yours. First, and foremost, many Agile practitioners seems to promote an idealized workplace where culture, practices & practices lead to total engagement & autonomy.
I’m slightly more confident that workplaces like that exist than I am in the existence of unicorns. Correction – small consultancies can and often do work like that. The kinds of places where these practitioners often work.
I see the ‘ceremonies’ of Agile as splints that can help the culture necessary for Agile to work grow.
More in a bit…
Dan Wiebe
I’d like to see this tried once:
First, the ops team gets called into a meeting by a director or VP. You have three months, says the suit, to learn how to put a new release into production once a week. You’ll have all the support you need, but at the end of that time, all our code will be rolled out every week. If you don’t have any new code, then roll out the old code again for practice.
Three months later, the dev teams are called into a meeting. Starting in one month, we’re going to start expecting to see new functionality in production every week. Doesn’t have to be much, but has to be something: and we’ll be watching.
Then every week the suit prowls the dev floor. “My dashboard says you people got the new loan-summary screen in last week. Good job! But you folks over here didn’t push anything. Why was that? Do we need to find a team who can?”
In a situation like this, it seems likely that the dev teams will discover and embrace Agile (and only the real kind, not anything ineffectual or fake) on their own in self-defense, rather than having it forced down their throats and being resentful.
They’d also insist on backlogs from their POs that enabled weekly releases. The culture wouldn’t blink into existence, but its flowering would be driven by what works on the ground rather than by touchy-feely theory or wishful thinking.
Charles Bradley
Mike, I know you and I have different views from time to time… but on this one, I think we’re in 110% agreement. Excellent post about how you need top down, bottom up, lean thinking, and learning new excellent technical practices.
The one thing I would add to everything you said, because again I’m philosophically in 1!0% agreement with you here, is that it’s vital to set up a set of meaningful outcome metrics before you begin, so that you and your client can co-evolve in a way that includes solid evidence that the change effort is working. It also discourages them from going back to old habits. I don’t want to be seen as pimping an approach that benefits myself, but let’s just say that if one looked at some really good evidence based approaches for software out there, you could easily find a great set of metrics to use to see if the client is actually reaping benefits. Of course, you need to level set with them that the real change and improvement has to come from them, not from you or your company. You can guide and “lead” as is stated in your company name, but you cannot do the heavy lifting for them.
Anyway, wish you well, and thanks for the excellent post.
Tom Henricksen
Mike, I couldn’t agree more this statement, “These kinds of changes have to be led with top down intent, but with a bottom up implementation. People have to be engaged and do ultimately have to be bought in.” In working at major career placement company I witnessed this first hand. We had to do an “Agile Transformation” twice after the first one stalled with no top down intent or engagement.
Thanks,
Tom
Daniel Mezick
Hi Mike,
Let’s say we “…have a leader that recognizes there is a problem and is open to alternative ways of solving it.”
Let’s say we get “…permission to start an agile transformation” and that this means I have permission to impose the “structure” suggested here.
Question: Is the ENGAGEMENT of the people doing the practices a NECESSARY PRECONDITION to being successful with this? (Yes or No)
Question: If ENGAGEMENT actually does matter, how is it encouraged by the imposition of Agile practices on teams without their consent?
I notice that Martin Fowler (a Manifesto signatory) strongly warns against this, in 2006 (see http://martinfowler.com/bliki/AgileImposition.html….)
“…“A team may choose a totally waterfall, un-agile process. In that case, clearly the process is no more agile than apples taste of strawberries. But agile methods aren’t the best for all situations, and personally I’d rather have a team work in a non-agile manner they chose themselves than have my favorite agile practices imposed upon them.” -Martin Fowler, Agile Manifesto signatory. Written 2006, the “Agile Imposition” blog post
My detailed commentary on that blog post can be found here:
http://newtechusa.net/agile/people-then-practices/
What do you say Mike?
Regards,
Daniel
http://www.DanielMezick.com
I’ve publicly stated many times that I do not believe every one has to be bought into doing agile for the organization to do agile. I think agile, specifically iterative and incremental, team based software development, where the team produces a working tested increment on some pre-determined cadence… absolutely should be mandated. I don’t believe that waterfall is ever the right approach to building software.
Within the confines of the business problem the team is hired to solve, and the boundaries of the broader organization they are empowered to work in, I think they should have a high degree of latitude to self-select whatever practices they want. Agile to me is not process, it is the intersection of organizational systems of delivery, practice, and culture. Our approach calls for starting with organizational systems, then practices, then culture. As always, happy to discuss our success stories and client list.
I am also fine going on record that I disagree with Martin Fowler on this point. I would never let an organization which was spending my money operate in a waterfall fashion. I don’t care if they think it’s the best way or not. Of course, you cannot make another human being do anything… but you can choose to make it a condition of their employment. They can always self-organize into another company.
If my answer sounds terse and short, it’s because you and I have had numerous back-channel conversations around this topic and you know where I stand. I think you are being provocative with your comment. This is the equivalent of asking a question such as ‘how long have you been beating your wife’ when you know full well I was not beating my wife. There is no answer to the question the way you’ve phrased it.
Happy to discuss further publicly or privately if you really want to understand what I think about this. If you want to have public debates to further your agenda, I’m not super interested in that conversation. Wish you all the best.
Nick Hughes
Hi Mike,
Great post, it really drives some clarity around the challenges faced by large enterprises trying to leverage Agile to obtain better delivery outcomes. In my experience its very dysfunctional and there is little understanding of exactly what ‘Agile’ is and where the improvements come from. Often we see a card-wall and a stand-up and magically a team is expected to delivery twice the work in half the time.
The lens of Culture, Practice and Structure is useful and I believe its chipping away at all three at the same time that gradually starts to move the mountain, the voice if the leadership is a key driver in how quickly that mountain moves…
I also think its worth trying to describe when we are NOT Agile and, while this can be tricky given different flavours and contexts I believe that the degree to which a team is empowered is a good yardstick. If a team cannot make any meaningful autonomous decisions about what the work is, or how it is delivered, you are not doing Agile.
This is the key challenge for large enterprise and one which few are seriously attempting to address. How to build trust, defer responsibility and surrender control to small independent cross functional teams. Its an antitheses of a culture that has been decades in the making in some cases.
Cheers,
Nick
Lisa Crispin
Given my limited experience with large companies, your recommended approach makes sense to me. I’d have thought the cultural change has to come first, but yeah, the structure has to be there, and cultural change could take a long time. I guess the cultural prerequisite that I see is that the leadership/management has to place value on delivering a quality product that will succeed over the long term. But I feel like our economic system, with the stock market, rewards short term success .
Putting a structure in place to do a better job of delivering business value frequently at a sustainable pace (my fave definition of Agile, from Elisabeth Hendrickson) is a major investment. Companies think “going agile” will make them “fast”. IME it takes years, not months, for teams to master the practices that allow them to be “agile”. If executives don’t care enough, or if they’re being rewarded for a double digit profit this quarter, they aren’t going to be willing to make that investment.
Sorry for the long lag time here Lisa. Just wanted to say thanks for the comment. We are going into some really large organizations that are really far from doing anything resembling small team agile. I believe that the next frontier in thinking here is how to refactor a legacy organization into an agile organization. The refactoring metaphor is powerful and holds on lots of levels. Thanks!