The Agile Revolution: From Waterfall to Manifesto to Industry

The Problem That Built the Waterfall

In the 1960s, software was written the way everything else in engineering was built: sequentially. You gathered requirements. You designed the system. You implemented the design. You tested the implementation. You delivered the product. This seemed obvious — it was how bridges were built, how aircraft were designed, how manufacturing processes were organized.

The analogy was wrong, and the wrongness took decades to fully understand.

Physical engineering operates on materials with known properties. The behavior of steel under load, the aerodynamics of a wing, the tensile strength of concrete — these can be measured, modeled, and predicted. A civil engineer who discovers during construction that a planned design is structurally unsound faces enormous costs: concrete must be broken, steel must be cut, timelines and budgets collapse. The sequential model makes sense precisely because late changes are catastrophically expensive.

Software has none of these constraints. Code is infinitely malleable: any line can be changed in seconds, any architecture can be revised if the team is willing to pay the time cost. But software also has a property that physical materials lack: its behavior cannot be fully specified in advance. The requirements that seemed complete at the start of a project reveal their incompleteness only when users interact with a working system. What customers say they want and what they actually want diverge in ways that become visible only through use.

This means that the sequential model fails software in a specific and predictable way: requirements locked at the start are wrong by the time implementation finishes. The testing phase, which comes at the end, is when problems with the fundamental design are first discovered — exactly the moment when fixing them is most expensive.

Winston Royce’s Misquoted Paper (1970)

The Waterfall model is commonly attributed to Winston Royce’s 1970 paper “Managing the Development of Large Software Systems,” published in the proceedings of IEEE WESCON. The attribution is ironic because Royce’s paper, read carefully, argued that the simple sequential model was dangerous.

Royce presented the sequential model in the paper’s second page — requirements, design, coding, testing, operations — and immediately followed it with this: “I believe in this concept, but the implementation described above is risky and invites failure.”

His actual argument was that software development required iteration — that feedback from later stages should flow back to earlier ones, that preliminary design should be followed by pilot implementation before committing to full design, that customer involvement throughout the process was essential. The paper included diagrams of iterative flows and feedback loops that bore little resemblance to the rigid sequential model that “Waterfall” came to denote.

What happened to Royce’s paper illustrates how methodology spreads: the diagram on page two — the sequential stages drawn as a cascade — was extracted, cited, and reproduced without the context of the pages that immediately qualified it. The most influential software methodology of the following three decades was built on a selective reading of a paper that warned against exactly what was being built.

The United States Department of Defense codified the sequential model as DOD-STD-2167 in 1985 and later MIL-STD-498 in 1994. For defense contractors, Waterfall was not merely a methodology but a contractual requirement: contracts specified deliverables corresponding to Waterfall phases, and payment was tied to phase completion. The methodology was baked into the financial and legal structure of large software procurement in a way that made it extremely difficult to change even as evidence of its failures mounted.

The Chaos Report and the Crisis

By the 1990s, software project failure was a recognized industry crisis.

The Standish Group’s CHAOS Report, first published in 1994, surveyed 365 companies about 8,380 software applications. The findings were stark: 31.1% of projects were canceled outright before completion. 52.7% of projects cost 189% of their original estimates. Only 16.2% of projects were completed on time and on budget.

The report defined three categories: successful projects (on time, on budget, with required features), challenged projects (completed but over budget, over time, or with reduced features), and impaired projects (canceled). The failure numbers were bad enough, but the deeper issue was that even “successful” projects often delivered software that users did not actually need: the Standish Group found that 45% of features in delivered software were never used, and another 19% were rarely used.

The software development field was not suffering from a shortage of talented programmers. It was suffering from a structural mismatch between its dominant methodology and the actual nature of software development. Requirements that were locked before implementation were wrong. Testing that happened after implementation was too late. Users who were consulted only at the beginning and end had no mechanism to correct course in between.

A generation of practitioners was developing alternatives to the Waterfall model. Their approaches shared common features: shorter cycles, closer collaboration with users, continuous testing, welcoming rather than resisting change. But they were scattered across different companies and communities, known by different names, and had not yet coalesced into a movement.

The Snowbird Weekend: Writing the Manifesto (February 2001)

In February 2001, seventeen software developers gathered at the Snowbird ski resort in the Wasatch Mountains of Utah. They had been brought together by Kent Beck, who had recently published Extreme Programming Explained, and the gathering included practitioners of various lightweight methodologies: Extreme Programming, Scrum, Feature-Driven Development, Dynamic Systems Development Method, Crystal, and others.

The participants were not strangers. Many had been circling the same ideas for years in different contexts — Jim Highsmith and Alistair Cockburn through organizational consulting, Kent Beck and Ward Cunningham through Extreme Programming, Ken Schwaber and Jeff Sutherland through Scrum. The Snowbird meeting was an opportunity to find their common ground.

Over two days of discussion, they produced what they called the Manifesto for Agile Software Development. The manifesto was deliberately minimal — a statement of values, not a methodology. Its core was four pairs of statements:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The phrasing was careful: “That is, while there is value in the items on the right, we value the items on the left more.” This was not an argument against documentation or plans. It was an argument about priorities when they conflicted.

The manifesto was accompanied by twelve principles that elaborated on the values: delivering working software frequently, welcoming changing requirements even late in development, close daily cooperation between business people and developers, building projects around motivated individuals, face-to-face conversation as the most effective communication, sustainable development pace, continuous attention to technical excellence, simplicity, self-organizing teams, and regular reflection and adaptation.

The seventeen signatories included figures who had been separately developing the ideas that the manifesto now unified: Kent Beck (Extreme Programming), Ken Schwaber and Jeff Sutherland (Scrum), Alistair Cockburn (Crystal), Martin Fowler (refactoring and software design patterns), Ward Cunningham (inventor of the wiki), and others.

The manifesto was published at agilemanifesto.org. It had no formal organization behind it, no certification body, no training curriculum. It was, deliberately, just a statement of values signed by seventeen people.

Extreme Programming: Technical Discipline as Methodology

Of the lightweight methodologies that fed into the Agile synthesis, Extreme Programming (XP) was the most technically prescriptive.

Kent Beck had developed XP while consulting at Chrysler in the mid-1990s on the Chrysler Comprehensive Compensation System — a payroll system project that had been struggling under conventional methodologies. Beck’s hypothesis was that if the practices known to produce good software were good, they should be applied to the extreme: if code review was good, review code constantly (pair programming). If testing was good, test all the time and test first (Test-Driven Development). If integration was good, integrate continuously. If simple design was good, always use the simplest design that works.

XP’s practices were mutually reinforcing. Pair programming meant that two developers worked at one keyboard, one typing (the “driver”) and one reviewing (the “navigator”), switching roles frequently. This was expensive in person-hours but produced code with fewer defects, spread knowledge across the team, and eliminated the knowledge silos that made individual developers indispensable and irreplaceable.

Test-Driven Development (TDD), which Beck had codified in 2002, inverted the conventional testing sequence. Instead of writing code and then writing tests to verify it, TDD required writing a failing test first, then writing the minimum code to make it pass, then refactoring the code for quality. The cycle — Red (failing test), Green (passing test), Refactor — became a discipline that embedded testing into the development process rather than appending it at the end.

The TDD cycle had effects beyond test coverage. Writing tests before code forced developers to think about how code would be used before thinking about how it would be implemented. This produced code with cleaner interfaces — because the tests were written from the caller’s perspective — and made the implicit assumptions in a design explicit in the test suite. It also produced a safety net for refactoring: if all tests passed before and after a change, the change preserved behavior, even if the internal structure changed significantly.

Continuous integration — integrating code from all developers into a shared baseline multiple times per day — was made practical by automated testing. Without tests, integration meant manually checking that changes hadn’t broken anything. With tests, integration meant running the test suite and trusting the result. Teams could merge frequently because the cost of discovering an integration problem was low (run the tests) and the cost of discovering it late was high (debug conflicts between weeks of divergent development).

XP’s social practices were as important as its technical ones. The on-site customer — a real product stakeholder embedded with the development team, available to answer questions and make decisions in real time — eliminated the information latency that made requirements drift so common. The planning game — a collaborative ceremony where developers estimated story cards and customers prioritized them — made tradeoffs explicit and put prioritization decisions with the people who understood business value.

Scrum: The Framework That Won

While XP offered a complete and technically specific methodology, Scrum offered something more adaptable: a lightweight framework for organizing work without prescribing specific technical practices.

Scrum was developed independently by Jeff Sutherland and Ken Schwaber in the early 1990s, drawing on a 1986 paper by Hirotaka Takeuchi and Ikujiro Nonaka in the Harvard Business Review titled “The New New Product Development Game.” Takeuchi and Nonaka observed that the most productive product development teams — at Honda, Canon, Hewlett-Packard — did not operate sequentially but in overlapping, self-organizing bursts of activity analogous to the scrum formation in rugby: the whole team engaging together rather than passing the ball in sequence.

Sutherland implemented the approach at Easel Corporation in 1993. Schwaber presented the methodology formally at OOPSLA (Object-Oriented Programming, Systems, Languages and Applications) in 1995. After meeting at OOPSLA, the two codified Scrum together.

Scrum’s core structure was the sprint: a fixed-length iteration, typically two to four weeks, at the end of which the team delivered a potentially shippable product increment. Work was organized in a product backlog — an ordered list of desired features maintained by the Product Owner, a role responsible for representing stakeholder interests and prioritizing work. Each sprint began with a sprint planning meeting where the team selected items from the backlog and committed to completing them. Each day included a short daily scrum — a fifteen-minute standup where each team member reported what they’d done, what they planned to do, and what was blocking them. Each sprint ended with a sprint review (demonstrating what was built) and a sprint retrospective (discussing how to improve the process).

The Scrum Master was a new role with no direct analogue in conventional project management: not a manager or a team lead, but a servant-leader responsible for removing obstacles to the team’s progress and protecting the team’s process from external interference.

Scrum’s power was its flexibility. Unlike XP, which required specific technical practices (pair programming, TDD), Scrum imposed only the sprint structure and ceremonies. A team could adopt Scrum without changing how they wrote code, which lowered the adoption barrier enormously. The framework could be applied to software development, hardware development, marketing campaigns, or organizational change — any complex work where the path forward was not fully knowable in advance.

This flexibility was also a limitation. Scrum without XP’s technical practices could easily produce fast, iterative delivery of poorly-designed code. Teams that ran sprints but lacked automated testing, continuous integration, or refactoring discipline often found themselves accumulating technical debt that eventually made the iterative pace unsustainable.

Kanban: Flow Over Time Boxes

While Scrum organized work into sprint time boxes, Kanban — adapted from Toyota’s manufacturing system by David Anderson starting around 2004 — organized work around flow.

The core Kanban insight was that work had a natural lead time — the interval between starting a task and completing it — and that this lead time was determined by the amount of work in progress. If a team had too many tasks in flight simultaneously, each task took longer because attention was divided and context-switching was constant. The Kanban solution was to limit work in progress explicitly: a column on a Kanban board might allow at most three items, forcing the team to complete current work before starting new work.

The Kanban board — columns representing stages of work (To Do, In Progress, Review, Done), cards representing individual tasks — became the visual management tool of the Agile movement, even in teams that did not formally practice Kanban. The board made work visible in a way that status reports and meeting agendas did not: anyone could look at the board and see where work was stuck, what was blocked, and whether the work in progress limit was being respected.

Kanban appealed particularly to operations teams and teams doing continuous work (bug fixing, maintenance, support) where the sprint cadence of Scrum fit awkwardly. You could not plan a two-week sprint of bug fixes when the bugs arrived unpredictably and had variable urgency. Kanban’s pull-based model — work was pulled into the next stage when capacity existed, rather than pushed through on a schedule — matched this kind of work better.

The Industrialization of Agile

The Agile Manifesto was a philosophical statement. It had no certification, no professional body, no standardized curriculum. By design, it was a set of values that different practitioners and teams would apply differently.

What followed, inevitably, was industrialization.

Scrum Alliance, founded in 2001 by Ken Schwaber and Mike Cohn, introduced the Certified Scrum Master (CSM) certification — a two-day training course that certified participants as qualified Scrum Masters. The certification was controversial from the beginning. Schwaber later resigned from the Scrum Alliance and founded Scrum.org, offering its own certification track, after disagreements about how the certifications were being sold and how the framework was being taught.

By 2010, Agile coaching and training had become a substantial industry. Companies offered multi-day certification courses, organizational transformation consulting, and coaching engagements. The Scrum certification ecosystem expanded to include Product Owner certifications, scaling certifications, and practitioner-level certifications. A practitioner could accumulate credentials — CSM, CSPO, CSP, CTC — that signaled Agile expertise to employers.

The certification industry served real needs: organizations adopting new methods needed training, and the certifications provided a common vocabulary and baseline understanding. But it also began to shift the center of gravity of the Agile world from practitioners solving problems to consultants selling solutions — the transformation of a value system into a product.

Dead End: SAFe and the Return of the Waterfall

The most commercially successful framework in enterprise software development is not Scrum. It is the Scaled Agile Framework, known as SAFe.

Dean Leffingwell developed SAFe starting around 2007, publishing the first formal version in 2011. SAFe addressed a genuine problem: Scrum and XP were designed for small teams, and most of the organizations that wanted to adopt Agile were large enterprises with hundreds of developers working on interconnected systems. How do you coordinate ten Scrum teams working on components that depend on each other?

SAFe’s answer was a layered hierarchy of teams, programs, and portfolios, each with their own ceremonies, roles, and cadences. At the team level, SAFe preserved the Scrum sprint. At the program level, teams were organized into Agile Release Trains (ARTs) — groups of 50–125 people that synchronized their work in Program Increments (PIs), typically ten-week planning cycles. At the portfolio level, strategy and investment were coordinated through a Portfolio Kanban tied to business outcomes.

The framework was comprehensive, well-documented, and certifiable. SAFe offered certifications for practitioners, consultants, architects, and transformation leaders. By 2020, SAFe was deployed in organizations across financial services, government, defense, telecommunications, and healthcare — exactly the large enterprise environments where waterfall methodologies had previously dominated.

The criticism of SAFe from within the Agile community has been consistent and pointed. Ron Jeffries, one of the seventeen signatories of the Agile Manifesto, wrote that SAFe “provides a convenient, if ineffective, way to do ‘Agile’ while barely changing how we work at all.” Alistair Cockburn, another signatory, described SAFe as “Agile theater” — the surface appearance of Agile practice without its substance.

The substantive objection is that SAFe reproduces at scale the problems that Agile was designed to solve. The Program Increment planning event — typically a two-day event where a hundred or more people align on what will be built over the next ten weeks — functionally reintroduces the long planning horizons and locked requirements of Waterfall. The Agile Release Train, with its synchronized cadences and dependencies managed through program boards, is a large-scale sequential coordination structure imposed on top of nominally Agile teams. The portfolio layer, which manages investment decisions and strategic themes, is conventional business planning with Agile vocabulary.

The teams within the structure can be genuinely agile — responsive to change, continuously delivering, collaborating closely with users. But the structure above them can — and often does — operate as a Waterfall program management layer. Requirements flow down from portfolio to program to team; deliverables flow up. The ceremonies are called different things, but the dynamics are familiar.

SAFe is not a dead end in the sense of a technology that failed to find users — it has found enormous commercial success. It is a dead end in the sense that it has reached a local maximum: an approach that is adoptable by large enterprises precisely because it requires less fundamental change than genuine Agile transformation, but that therefore does not achieve the organizational responsiveness that the Agile Manifesto promised. SAFe organizations often report that they have “gone Agile” while their developers report that the fundamental dysfunctions — late-discovered requirements, coordination overhead, difficulty responding to change — remain.

The Agile Manifesto’s authors saw this coming. The manifesto’s twelfth principle — “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly” — was meant to be a mechanism for continuous improvement. In organizations where the improvement retrospective is a meeting that produces action items that are never implemented, the mechanism does not function.

The Lasting Contribution

What has Agile actually changed?

The most concrete legacy is the sprint and the daily standup. Teams that do not call themselves Agile, that have never read the manifesto, now organize their work in iterations and coordinate through brief daily meetings. The vocabulary of backlogs, user stories, and retrospectives has entered mainstream software development practice. The idea that software should be delivered in small, frequent increments rather than in a single large release has become so obvious in commercial software development that it barely requires argument.

The practice of Test-Driven Development, which XP pioneered, is now a professional norm in many development communities — not universal, but recognized as a mark of craft. Continuous integration, continuous delivery, and the DevOps movement that followed are direct descendants of XP’s technical practices.

The Agile Manifesto also changed how software organizations think about change. The waterfall premise that requirements could be locked in advance and that deviation from the plan represented failure has been largely discredited. The alternative premise — that requirements will change, that uncertainty is normal, that the appropriate response is to structure work so that change can be accommodated — is now broadly accepted, even in organizations that do not formally practice any Agile method.

What the movement did not change is the organizational and political structures that make genuine agility difficult in large enterprises. A small team with direct access to users, authority to make decisions, and accountability for outcomes can be genuinely agile. A team of developers embedded in a large organization, receiving requirements from a product organization that answers to executives who report to a board with quarterly earnings expectations, faces structural forces that push against responsiveness and iteration regardless of which methodology the team adopts.

The manifesto’s values are as relevant as they were in 2001. The gap between those values and most organizations’ practice remains roughly as wide. The history of Agile is in part a history of a good idea encountering organizational gravity — and the organizational gravity winning more often than its opponents like to admit.

For the infrastructure practices that emerged from Agile’s technical heritage, see The Open Source Revolution and The Rise of Version Control. For the organizational movement that attempted to bridge Agile development with operations, see the forthcoming article on the DevOps Revolution.

---

📚 Sources

• Beck, Kent, et al.: Manifesto for Agile Software Development (2001) — agilemanifesto.org
• Beck, Kent: Extreme Programming Explained: Embrace Change, 1st Edition (1999), Addison-Wesley
• Beck, Kent: Test-Driven Development: By Example (2002), Addison-Wesley
• Schwaber, Ken & Sutherland, Jeff: The Scrum Guide (2020 version) — scrumguides.org
• Royce, Winston W.: “Managing the Development of Large Software Systems” — IEEE WESCON Proceedings (1970)
• Standish Group: CHAOS Report 1994
• Takeuchi, Hirotaka & Nonaka, Ikujiro: “The New New Product Development Game” — Harvard Business Review, January–February 1986
• Fowler, Martin: Refactoring: Improving the Design of Existing Code, 2nd Edition (2018), Addison-Wesley
• Anderson, David J.: Kanban: Successful Evolutionary Change for Your Technology Business (2010), Blue Hole Press
• Leffingwell, Dean: Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise (2011), Addison-Wesley
• Jeffries, Ron: “Developers Should Abandon Agile” — ronjeffries.com (May 2018)