Why is Agile so Hard to Sell?
What I find incredibly interesting is why defining value is so hard. Agile proponents have been beating the value drum since the very beginning. Put the customer in the room… understand their needs… build what ever they want… deliver software in small increments… get constant feedback… converge on the optimal solution… deliver value early and often. Agile is all about delivering value. Why wouldn’t a management team embrace a set of methodologies so focused on giving them what they need the most?
Here is my take…
Agile is (in large part) a reaction to misapplied waterfall development and naive application of project management principles in ways that are inconsistent with how software actually gets built. It was is a reaction to dehumanizing, process and artifact driven management approaches… processes that assumed with enough procedures, we could somehow commoditize the practice of software engineering. We wanted to take the uncertainty out of a craft that is really a blend of engineering and art. Our desire was to make everything predictable and repeatable.
We were trying to treat people like machines that could be infinitely sliced across tasks, tasks that with the right level of process and planning, with the right level of project management and oversight, would somehow magically roll up into working software that our customers would want to buy. Guys like Jefferies, Cockburn, Schwaber, and Sutherland were beginning to realize that successful projects were really the result of high-performing, committed individuals, working together in small teams, all directed toward shared outcomes.
XP, Crystal, and Scrum were born through the realization that these small empowered teams delivered the best outcomes and were the most successful over time. As these agile approaches were beginning to emerge, they all shared this disposition toward small teams, close customer interaction, and frequent delivery of value. The funny thing is that if you talk with most folks working for small companies… this is kind of what they do naturally. Folks are not assigned to a single role… you have a team of high-performing individuals working together for the sake of their collective survival. Big companies seem to have messed this up… but I digress.
Fast forward 10 or 15 years…
Now we have pockets of agile within large organizations… sometimes we might even have agile across entire large enterprises. We are exploring agile methods and trying to see if they can deliver on the small team promise… but in the large. The main difference with these larger organizations is that value isn’t the same as it is in a small team or a small company. There is not a direct correlation between team performance and business outcomes… there is not a direct connection between what the team delivers and what we can sell to our customers. It takes the output of too many small teams coming together to deliver anything of value.
Agile methodologies have typically treated the team like a black box. We give them a prioritized product backlog as an input and we get valuable working software as an output. But now… we have to coordinate the output of several teams… maybe even dozens of teams… into some sort of coordinated deliverable. We are having to deal with getting tens or hundreds of people working together in a coordinated fashion. When that is our context… the message of teamwork and collaboration… close relationships between the team and the customer doesn’t resonate. The only way many of these organizations can get any sense of comfort… any sense they that they are responsibly managing their part of the business… is to ask their teams to commit to the big up front plan
As an organization, we know that we need to deliver value as fast as possible. We know that we need to be able to respond to change. We know that we need to deliver with more predictability and with higher quality. We have an intuitive sense that what we are doing isn’t working. We want to get benefits that agile talks about. We suspect that something like Scrum or XP can help… but we can’t figure out how to apply the small team concepts to our particular business problems. That’s why you get the classic “agile will never work here ” comments. There is an inherent disconnect between the team level guidance agile methodologies talk about and the bigger concerns your senior executives are struggling with.
There is a gap between value at the team level and value at the enterprise level.
At the end of the day, our community needs to develop guidance that helps our executives get the benefits of agile but focuses on real, enterprise class value delivery… value defined in terms of real results and real business outcomes. So, why is agile a tough sell? We are asking our leaders to trust a process that focuses on team based outcomes but doesn’t give them a credible way to articulate how the development organization is going to deliver on the more complex objectives of the business.
Comments (35)
Federico Zuppa
Great post. It is still an opened topic how Agile scales to the large enterprise, right? (I've seen a few books, like this one http://bit.ly/2GvBB, but haven't read it yet). One thing that is true is that as the project gets more complex and there is more people involved, you need more processes to coordinate the work and thus it becomes less agile (there is a matrix in Cockburn's book that shows this).
Scott Duncan
I'm not convinced Agile has to solve "world hunger," i.e., all aspects of value fromthe bottom to the top of the enterprize, to have validity and usefulness in the organization. I certainly don't see a phased, sequential development lifecycle solving this whole value spectrum issue either. So I do not believe that is the issue, thopugh the issue may be related.
In the past, whenever some new process approach has arisen, it has been my experience that there are those who will, for one reason or the other, require that it solve "world hunger" to prove itself. This is a wonderful way to stave off change, of course.
It isn't that blatant, of course. Usually it is couched in requests to show the ROI for change or some such thing. Now juist about anyone in a larger organization knows what a red herring this sort of request is. Most organizations do not have a reliable eough data collection and analysis effort to "prove" anything except what people already want to believe.
Some 15-20 years ago, the demand was to prove the ROI for process improvement efforts. People still argue this, of course. But I am reminded of what Bill Curtis once said about ROI measures for PI. By the time an organization has data reliable enough to really answer this question, they no longer seem to demand the proof. Their efforts to get them to where they have such management by data have already proven PI's value to them.
It's probably the same of Agile (as any other such change), i.e.,. by the time an organization can answer the question, they likely don't ask it any longer.
Pawel Brodzinski
I think you pretty much hit the nail. I was talking recently with a guy from company heavily employing Scrum and selling it out as their differentiator. They were very successful at that and I was all about "how the heck you're doing that?"
Then he told us they're doing very small projects (2-3 iterations) mostly. And of course they're small organization.
When I try to map agile to big organizations I usually end up with some agile teams usually working with Scrum but covered with old-school heavy-weight project management process. Teams have their long-term goals and agile is just the way one or another team responds to their commitment. On the high-level it's just a small piece and it doesn't really matter whether they employ Scrum, waterfall or hardcore hacking.
FlowChainSensei
Excellent post!
I tweeted a similar observation the other day: "Is Agile without Systems thinking a train wreck?".
Agile is most often a local optimisation stemming from a desire to address the immediate concerns of smart developers who could see there was a better way than the old ways. But being a local optimisation, inherently it can only go so far towards contributing to better business outcomes. Most times, this is not far enough. Hence the current interest in Systems Thinking, Lean, Theory of Constraints, etc.
Unfortunately, many developers do not have much leverage outside their own teams to get these ideas into the mainstream in their organisations.
Matthias Marschall
I agree that it is difficult to apply team centered processes across the enterprise. But valuing individuals and their contribution would definitely help. Another critical point is to try to keep teams together and assign work to _them_ rather than "staffing projects". Both things could help to make things work better in the enterprise.
Mike Cottmeyer
Federico,
I have liked Cockburn's grid for quite a while now. Agile fits in the lower left quadrant. Many organizations are trying to apply it to the top and far right quadrants. I think that can be done, but there are additional processes and frameworks that need to be put around agile to make it successful in these larger, more mission critical projects. What I am trying to draw attention to is that small-team-agile by itself doesn't scale and the value message as articulated by small-team-agile doesn't always resonate with those that hold the purse strings. Thanks for the comment.
Mike
Mike Cottmeyer
Scott,
Sure… agile doesn't have to solve world hunger or offer a solution to the Middle East peace process. But I get asked all the time… how do I sell agile to my leadership team? People realize that Scrum at the team level… within a larger enterprise… is a local optimization. These teams have success and then they get busted up and redistributed, only to have to rebuild what they already had.
So sure… if you want to do Scrum at a single team level and not boil the ocean… go for it. Just don't be surprised about how all that 'value' you are creating working with the 'customer' falls on deaf ears the next time your organization decides to re-org.
Not trying to be snippy, I promise ;-) but when we focus on only getting better within our own problem domain, sometimes we actually do damage to the overall system's ability to be productive.
Thanks for the comment!
Mike
Mike Cottmeyer
FlowChain,
Can I assume that Sensei is your last name ;-)
Thanks for the comment. As a development team or a project manager, you are correct, most often we don't have the influence to lead the kind of change necessary in our organizations to make agile sustainable. I think there are enough execs paying attention now, that maybe… just maybe.. if we keep talking about this issue… the next time they decide to re-org, they'll take some of these issues into consideration.
If nothing else, maybe we are educating the next generation of organizational leadership.
Thanks for the comment!
Mike
Mike Cottmeyer
Matthias,
Yes, valuing individuals is important as is keeping teams together. That said… most organization have structures and beliefs that work against these goals every day. HR policies, compensation plans, career planning, resource tracking, and resource capacity reports all drive our business to try to optimize for individual performance and individual utilization over team based or performance or team based outcomes.
So while you are correct, not taking a systems approach to organizational design, will result in long term failure. IMO… sorry to sound so dire ;-)
Derek Huether
Mike, excellent post! It really made me think.
Though I have seen the desire of an enterprise management team to embrace Agile, allowing more value to be delivered, I have also seen them take pause and throw up roadblocks. At the moment they believed the top-down command and control structure would be weakened by bottom-up empowered teams, delivering value suddenly didn't appear to be as important. Though I appreciate the necessity of enterprise management teams to provide strategic vision, I believe tactical implementation should be left to those outside their group. The hierarchy of wants, not needs, for the management team differs from the implementation team, if we want to admit it or not.
The key question asked is why wouldn't a management team embrace a set of methodologies so focused on giving them what they need the most? My answer is I believe it is because delivering value is NOT necessarily what they WANT, it is what they NEED.
Mike Cottmeyer
Interesting observation Derek. You and I have talked about this recently. The goal might not actually be to deliver more value. It might be to cover yourself, spend money at the right rate, make sure you don't rock the boat, get that promotion or bonus. You'd like to think that these things are correlated to value delivery, but you know as well as I do… that isn't always the case.
Great comment!
Mike
Kenneth Clyne
Good post Mike. As I work with companies trying to scale I find that agile is the wrong word to have in my lexicon. Many in senior management have already made up their mind what agile means and I can't change that perception (at least not in the short-term). Better to focus on defining and delivering value this gets straight to the bottom line.
Anonymous
You very succintly captured what my organization is struggling with right now. It was small company that grew quite quickly and a few years later is quite large. They are struggling with many of the things you have covered in this post. I would love for you to post ideas on how to solve some of these challenges. What are the next steps?
Mike Cottmeyer
Kenneth,
Thanks much for the comment. I've explored this point around agile being a means to and end… it is the end that counts… and the fact that most folks don't really care about agile. Take a look at this post of you missed it.
https://www.leadingagile.com/2009/12/adopting-agile-isnt-point.html
Mike
Mike Cottmeyer
Anonymous,
i've been exploring the solution to this on Leading Agile for the last year. I am restarting with the problem statement just to try to get some clarity. As I move forward and talk to more people, my ability to articulate the problem hopefully) gets better. I'm also writing a book around this, so some of these posts are exploring language for the first few chapters.
That said… take a look at this video I did for VersionOne. You'll have to register with them to get the feed, but I think it is worth it. It is one of my early attempts to explain the value over velocity problem.
https://www.leadingagile.com/2009/08/adopting-agile-video.html
Bob Williams
Mike,
Excellent post.
People resist change and a management team will resist change because it poses a risk to their organizational power and standing. This is especially true if you go into an organization and tell them that the way they are doing things now is wrong and you have a solution to their problem.
I don't sell Agile for a living, but as an outside observer I wonder if you should strip the names from your recommendations. Rather than telling a management team you have a new program (Agile Development) for them to embrace truly focus on the business value and need. Don't tell them they are doing it wrong, but show them you have incremental improvements to make their organization more customer focused. Treat the sales engagement and organizational transition much like the methodology itself. Small iterative steps rather than a wholesale change.
Regards,
Bob
Mike Cottmeyer
Man I have a lot of smart people that read my blog… really, really cool!
Great comment Bob. I believe in incremental improvement. We have to focus our improvement efforts on the organizational areas that deliver the most value as quickly as possible. I keep saying this… it is NEVER about agile. It is always about delivering value as quickly as possible.
Mike
Thierry
It's all about money and the small increments at the end of the agile process scares clients. It's like a never ending story.
What clients like in waterfall is that they are no turning back once they have validated what they want it's no longer there problem. It's the developer's team problem.
In agile, clients are massively involved they have to say what they want and they have to assist meetings every week to monitor how developments goes, worst, during developments you are going to unveil some black spot they don't want to speak about.
That's why basic clients don't like agile, an agile project confront them with there greatest fear, the fact that sometimes they don't know what they want and they want you to find the best solution and they want to be able to blame you for having done the wrong choice.
Sometimes the first priority of clients is not having the perfect product matching exactly what they truly want and need. No, what clients want is not to be blame in case of failure.
Mike Cottmeyer
Theirry,
I don't disagree that you have a valid point in some contexts. I just don't necessarily think your point is universal. Some clients I work with like having the control over their product and like being in the driver's seat regarding project outcomes. That said, many organizations, and many product owners, just aren't setup to have full time product owners engaged with the team. Many don't have the knowledge to interact at the level the team needs them to interact. Guiding the team is not their passion.
In my more cynical moments, it is easy to assume that folks don't want to be engaged with the team because they want someone to blame… and while that might be true some of the time… I sure hope it's not universal.
Thanks for your comment.
Mike
David Bland
Mike,
It isn't a coincidence that the tone of our Agile conversations include more Lean, Kanban, and Scrumban topics.
I beleive it is happening organically because:
1. The Agile community is full of smart people.
2. We are looking for solutions to bridge this gap.
3. More and more enterprises are experimenting with incremental delivery in a down turn ecomony.
I applaud your efforts, and anticipate that these Agile/Lean threads will grow and evolve.
-David
Mike Cottmeyer
Thanks David,
I agree that there are many of us out there looking for ways to solve this problem and preserve what is good about agile. When I get preachy, or do some sort of call to action, I am trying to make sure people understand this is a problem. I am not sure everyone recognizes this yet. I also want to draw attention to the fact that simple solutions might not always address our complex situations. We have to be very contextually aware.
Good hearing from you, thanks!
Mike
Scott Duncan
You finish up by saying that Agile is a "tough sell" because "We are asking our leaders to trust a process that focuses on team based outcomes but doesn't give them a credible way to articulate how the development organization is going to deliver on the more complex objectives of the business."
And several times you note that a small team agile approach does not scale to the real issues of a large organization. In comments you note that to move over and up Alistair's matrix requires "additional processes and frameworks that need to be put around agile to make it successful in these larger, more mission critical projects…."
You also note in comments about how some large clients/projects want to retain control, not provide the full-time PO support, etc. That is, they are not interested in/able to follow an Agile approach. Thus, selling Agile in such circumstances is likely to fail, i.e., be "a hard sell."
All this suggests to me that your main point is how some situations are not appropriate for taking an Agile approach. Trying to force-feed an Agile approach in these cases will not succeed. Some other approach(es) are needed, or, at least, an Agile approach that doesn't require specific practices inappropriate for the situation.
Is this, fundamentally, the message?
Mike Cottmeyer
Thanks for the thoughtful comment Scott.
No… the fundamental message isn't that agile won't work in these environments. My message is that we need to expand and broaden our definition of agile. I've been to too many customer sites where agile = scrum.
If I reframe your assessment to ask if I am suggesting there are places where scrum is not the answer… I would say an emphatic *YES*!
I am a big tent agile guy… we need to embrace more than Scrum, because while Scrum is powerful, it does not tell the whole story.
Scott Duncan
Okay…then the post is about Scrum's limitations, not Agile's, in addressing the larger picture. What about XP and Crystal which you mention when you also mention Scrum? And does the concern extend to FDD and DSDM as other Agile methodology examples? I realize these are less widely employed/promoted, but, esp. for DSDM, the target seems to have been larger orgs and project sizes.
So what Agile methods/practices do we need, more than Scrum? Or is it comething else, like Lean/Kanban?
Mike Cottmeyer
Scott,
Not really. It is more about the fact that we tend to lose sight of what is really valuable to our businesses. Is a user story an increment of value? To a developer yes, do a Product Manager, no. To a senior VP of Product Management… these guys doing want to think about anything less than than 3-6 months of product deliverables… in aggregate.
Any of the agile methodologies can suffer from this. Scrum has the most mindshare right now. XP is pretty much in the same boat being disconnected from value. The same for Crystal. AUP and DSDM are better extended and recognize the business stakeholder more formally. I think since AUP has more of a solid tradition in RUP and Use Case Analysis… it is better positioned to think bigger.
Anyway… I think the solution involves elements from all of these. Scrum, XP, Crystal at the team level. AUP and DSDM as we try to scale. Lean as we think about value flows across the enterprise. I am writing my book around this topic, so needless to say, the solution doesn't lend itself to a single blog post. And if you read the next post, I explain that the post was intended to clarify the problem, no so much articulate a solution.
Hope this helps, somehow I am guessing not ;-)
James
Hey Mike! I'm late to this string, but I thought I'd pitch my two cents. I think one reason it is hard to sell agile into larger environments is rooted in the same reason these organizations trended over the past decades to heavy methodologies in the first place – fear. Fear of uncertainty. I would assert that software development is intrinsicly uncertain and that we humans are equally intrinsicly fearful of uncertainty. We like to *know* what the outcome of any action will be. We particularly want to *know* the return on our investments (the larger the investment, the greater this need to know). So, we've tried to create processes and controls that will increase our certainty. Of course, this has generally been unsuccessful, but (ironicly, perhaps) we are somehow more comfortable with the belief that we did all that we could even in the face of evidence that it is ineffective (at least it was *something*, I suppose).
There are a lot of advantages to agile approaches, but one of the strongest (in my opinion) in the large is the delivery of tangible product earlier and more frequently. Even if this doesn't accelerate the process (I believe it does) it still provides positive feedback that we're on the right path, data to justify (or adjust) our assumptions, and the opportunity to more quickly adapt to changing conditions. But, in some ways it also embraces that uncertainty that is so unsettling. Of course, this is one of its strengths. But acknowledging this and accepting it is very difficult. Even when we're bold and take some steps, we're still frequently dragging that security blanket along with us – just in case :)
Stan Yanakiev
Great post, Mike! This is the best analysis of why Agile is attractive at team level but hardly flies at a higher level.
Steve Denning
Hi Mike,
The problem is not so much that Agile doesn’t scale. The problem is rather that the whole dynamic of Agile in terms of delivering value to customers is incompatible with traditional management which is aimed fundamentally at making money for shareholders.
Agile makes money by delivering value: making money is a consequence not the goal. Traditional management sets out with the goal of making money and ends up exploiting customers and employees as needed to achieve the goal. These are two very different philosophies and they lead to radically different management practices. When the two sets of management practices come into contact with each other, traditional management tends to crush Agile, whether on a small scale or a large scale.
The good news is that there is now a widespread recognition that traditional management is failing even on its own terms. It is becoming less and less productive. It doesn’t fit today’s marketplace or today’s workplace. There is a now a fairly large-scale movement to reinvent traditional management in a way that represents an evolution of Agile values, i.e. a focus on delivering value to customers. I have described this evolution in a series of recent blog posts beginning here: http://stevedenning.typepad.com/steve_denning/2010/11/the-deathand-reinventionof-management-a-draft-synthesis.html.
I have also discussed the revolution under way in more detail in my new book, The Leader’s Guide to Radical Management: Reinventing the Workplace for the 21st Century (Jossey-Bass, 2010).
The reason that Agile is hard to explain to traditional managers is thus that it represents a radically different philosophy. I have sketched out in a blog post in a blog post how Agile CAN be explained to a CFO: http://stevedenning.typepad.com/steve_denning/2010/09/how-do-you-explain-agile-or-radical-management-to-a-cfo-.html
More fundamentally the need is to explain to traditional managers that their way of managing is an increasingly anachronistic. The exemplars of the new management philosophy like Apple, Amazon and Zappos are so successful in the marketplace that the continued evolution in this direction is inexorable. The only question is how rapidly it will happen.
Albert Lewis
I think and still maintain that Agile and Waterfall are fundamentally different because they address fundamentally different problems/situations. Agile arose to address situations with high uncertainty around requirements yet there was a need of something functional in very short periods of time…hence the notion of gradually “discovering” requirements, a f ew at a time, and developing/delivering software code to address what was known at the time and rapidly iterating to gradually increase the functionality addressed. Waterfall, addresses situations where there is room for time to dsicover the requirements in a more holistic manner and think through their implications prior to design/development. each of these methods have logical consequences in terms of scope creep/change control.
Additionally, Agile works (and not just works best) when you have small teams, where each individual is of high calibre and experience with a strong positive team chemistry and energy. Without this, your agile development will not really be agile and there will be huge amounts of rework (as discover of newer requirements increasingly requires design rework) due to designers not being of sufficiently high calibre.
Hence I do not believe that Agile can ever fully substitute for Waterfall, nor vice versa.
Lewis, yours is an interesting comment. I agree with some of your points and disagree with others.
Rather than try to untangle it, I’ll just leave you with this:
Regardless if I have time or not to discover… I have to weigh the knowability of the problem. I have to evaluate my assumptions about certainty. If the problem is knowable and there is no incremental value to be gained by delivering parts of the solution early, waterfall is fine (maybe). If I can reduce risk through early delivery (business risk, technical risk, logistical risk) and optimize my business outcomes, then I might want to consider agile.
I don’t buy that agile is only about emergent requirements. I believe that agile is nothing more than a risk reduction strategy. Reduce the risk I am building the wrong product, reduce the risk that I have a solution which will meet my time and cost constraints, reduce the risk I can actually build the product, reduce the risk that I’ll run out of funding, reduce the risk that I am building poor quality or am out of sync with the rest of the delivery team.
Waterfall tires to mitigate risk through documentation and sign-off. Agile reduces risk by incrementally building the product. To me, it’s as simple as that… but at the same time, as I started replying to this, I realized that I could go on forever about how waterfall is a suboptimal delivery strategy on just about every possible software project I could think of. Anyway… just my $0.02.
Albert Lewis
Dont disagree with you, nor does it I think shift my point. I think the difference is whose point of view you take. If you take the customer’s point of view where your view or mine, both would be valid simultaneously.
However if I was an outside contracting software development firm (say), then if a customer came to me and said ‘Agile’ (or equivalently, the situation called for agile), I would respond with ‘Time & Materials’ but if he said ‘Waterfall’, I would perhaps be willing to bid ‘Fixed Price” especially if i was fully confident about the level of my process capability/maturity. My view would be controlling the risk of ‘runaway scope creep’.
Paul Ionica
I will agree with most other commentators, this was a great and refreshing post.
I appreciated the summary of the crucial differences between traditional management techniques and agile, and that the driving force behind the agile mentality is deliver customer value.All nice and tidy there.
But what I valued even more was your honesty in addressing plainly the challenges that agile encounters when it is implemented in larger organizations or at an enterprise level. It may take some humility to acknowledge the current limitations of agile and where it currently struggles to gain a footing. Yet this may be the first step in developing long term solutions for enterprise wide implementation of agile.
Keep up the good work!
Albert Lewis
Now nearly as old as Uncle Bob Martin, but being only recently introduced to his talks, I have just watched a few of his videos.
so now, I have reason to pause and reconsider my position on Agile.
I would mostly still stand by my comments in 2011,2015 on agile, I would still stand by. I also may stand corrected that perhaps with Agile, the customer gets to see more deliverables at shorter cycles of times and that may overall itself be a plus to the engineering process. I think I now understand better why these guys set up the Agile Manifesto. Even if you throw out all the immature crap the testosterone driven youngsters talk up around agile, but keep short iteration cycles, small highly competent teams, good articulation and communication, good training in software engineering discipline AND keep that original Agile Manifesto up front and centre, then I would agree Agile can work very well, both for the programmers, for the project manager and for the client.
But is that like saying any group of highly competent disciplined well trained engineers will always outclass most other formations ….?
And if like Uncle Bob said, every 5 years, the number of programmers double and half of the programmer population have experience < 5 years, ….then the discipline that’s necessary and the level of skill and communication and maturity necessary is more likely than not going to be 180 degrees against the spirit of the Agile Manifesto?