42% of Agile projects are successful. Only 13% for the traditional Waterfall approach. This data comes from the CHAOS Report 2020, based on 50,000 projects.
Agile. Is it a magic word? Are sprints and daily planning the key to success? Is a startup that delivers 10 times a day in production Agile? Is it agile for teams to draw up two-day PI plannings to plan for three months of work?
Recently, I asked myself, “What is agility? Would I have worked in so-called “Agile” teams for several years without understanding the concept? Yes… Lost in this cacophony, I decided to go back to basics and try to understand the original intention of agility. Here’s a summary of what I learned.
The comparisons made below are mainly taken from the references cited at the end of the page. Many thanks to Erwan BOURGEOIS (Agile Coach) for his help in writing this post!
A brief history
Historically, engineers have created physical structures: bridges, roads, tunnels, machines… The manufacture of these structures requires several development phases, including design and construction. For several months, people work on drawing up specifications. They define needs, constraints, stakeholders, and budgets and draw up a realization plan. There’s little room for error, so engineers try to anticipate and predict everything. In these projects, the need is generally clearly expressed by the customer. In the 1970s, a new category of work emerged: software. Teams of engineers transposed the organizational know-how of traditional projects into software development. Softwares are designed, produced, and delivered to customers on floppy disks. No room for error, and no Internet to update softwares.
Over time, softwares became more complex. The Internet appeared and the number of customers grew. Corporate demand for software exploded. It drastically increased the number of software products developed each year.
The Standish Group CHAOS Report
In the mid-1990s, a firm called Standish Group decided to conduct a study of software development in the USA. They interviewed 365 IT managers from companies of all sizes. The sample consisted of 8,300 projects. Result: 16.2% completed software projects on time and on budget! 52.7% were over budget, over time, or delivered with a reduced scope. Finally, 31.1% of projects were abandoned.
The Standish Group teams estimated that in 1995, US companies and government agencies had spent $81 billion on canceled projects. The study report provides details of the results. Including the causes of these failures. The conclusion is interesting:
Application software projects are truly in troubled waters. […] Research at The Standish Group also indicates that smaller time frames, with delivery of software components early and often will increase the success rate. Shorter time frames result in an iterative process of design, prototype, develop, test and deploy small elements. This process is known as “growing” software, […]. Growing software engages the user earlier, each component has an owner or a small set of owners, and expectations are realistically set.
During this period, several groups of experts realized that building software like building a bridge didn’t work. These engineers invented new methods (lightweights), better adapted and oriented towards “Growing software”: SCRUM, Extreme Programming, Crystal… Conclusion? Software development is not predictable.
Among other things, they note:
- coding software costs less than building a bridge. So the software design phase is less profitable. Whereas a bridge requires 15% of the budget for design, for software this may exceed 50%.
- the success of a project depends on the people who make up the development teams: their relationships, and their creativity.
- it’s difficult to plan an entire software development phase in advance. Needs change frequently. Technologies evolve rapidly. Some needs only become apparent once the product is in the hands of the customer. If everything has been planned over long periods, modifications are very costly.
- Finally, separating the various players creates information silos (marketing, analysts, designers, developers, and others). This creates communication barriers. Which is not good for the product, and therefore not good for end-users.
Agile Manifesto
In 2001, 17 of these experts met in a ski chalet to try and find a common abstract basis for their different methods. After two days of work, they sketched out the Agile manifesto containing four values. In the months that followed, group members added twelve principles.
In my opinion, these two sentences from Martin Fowler sum up the foundations of Agile very well:
Agile methods are adaptive rather than predictive. Agile methods are people-oriented rather than process-oriented.
Martin Fowler
To predict is “to announce that a future thing will happen”, not to announce a future thing that will happen. If the prediction is wrong, the plan falls “into troubled waters”. To adapt is to “adjust one thing to another”. To adjust projects, organizations, or code to the vagaries of the environment.
Sofware is a compound word. The word “ware” means “product”. The word “soft” means easy to change. Therefore, software is a product that is easy to change. Software was invented because we wanted a way to quickly and easily change the behavior of our machines.
CleanAgile - Uncle Bob Martin.
Agile software development means being adaptive, not predictive. We can’t predict (cf. CHAOS Report), so let’s try to react to events as best we can. How can we do this?
By valuing change and minimizing costly processes.
By listening to user feedback.
By testing applications as often as possible. Through frequent incremental releases.
By building teams of autonomous people capable of self-organization and rapid adaptation.
In short, anything that enables us to be: adaptive rather than predictive, people-oriented rather than process-oriented.
None of this is possible without a certain level of technical excellence. That’s why, as developers, we’re key players in the agility movement. Our technical solutions make agility possible. And it’s through our actions and the way we develop that we can implement it.