Skip to content

Famous Software Disasters

Abstract

Software fails differently from bridges. A bridge that collapses was overloaded or badly built; a program that kills people usually did exactly what it was written to do. The canonical software disasters, the Therac-25 radiation overdoses, the Ariane 5 explosion, the Patriot missile timing drift, the Mars Climate Orbiter unit mix-up, Knight Capital’s $460 million in 45 minutes, and the UK Post Office Horizon scandal, are studied because each one exposes a failure mode that still exists: reused code in a new context, removed safety interlocks, accumulated rounding error, mismatched assumptions between teams, untested deployment procedures, and institutions that trusted computer output over human testimony.

Therac-25: When Software Replaced the Interlock

The Therac-25 was a radiation therapy machine built by Atomic Energy of Canada Limited (AECL). Its predecessors, the Therac-6 and Therac-20, had hardware interlocks that physically prevented the electron beam from firing at high power without the beam-spreading target in place. In the Therac-25, those hardware interlocks were removed; safety was enforced by software alone.

Between June 1985 and January 1987, the machine massively overdosed six patients in the US and Canada. At least three died from the overdoses. The principal defect was a race condition: if the operator entered and corrected treatment data quickly enough (something experienced operators did routinely) the machine could fire the full-power electron beam without the target in place, delivering doses estimated at over 100 times the intended amount. The machine reported only a cryptic “MALFUNCTION 54,” and operators, conditioned to frequent benign error messages, resumed treatment.

Nancy Leveson and Clark Turner’s investigation, published in IEEE Computer in 1993, became the founding case study of software safety engineering. Their conclusions reached beyond the bug: AECL had reused code from the older machines without systematic review, had no independent software testing, performed no failure analysis that included software, and repeatedly assured users the machine could not overdose. The deeper failures were institutional; see Secure by Design: A History of Software Security Engineering for the parallel history in security.

Ariane 5 Flight 501: Reused Code, New Rocket

On June 4, 1996, the first Ariane 5 rocket veered off course and self-destructed about 40 seconds after liftoff from Kourou, French Guiana, destroying the four Cluster science satellites it carried, a loss commonly put at several hundred million dollars.

The inquiry board’s report (the Lions report) traced the failure to a single software operation: a conversion of a 64-bit floating-point value (the rocket’s horizontal velocity) into a 16-bit signed integer. The code had been written for the Ariane 4, whose flight profile kept the value within range. Ariane 5 accelerated faster; the value overflowed; the inertial reference system shut down and emitted a diagnostic bit pattern that the flight computer interpreted as flight data. The identical backup unit, running identical software, had failed identically 72 milliseconds earlier. The function that overflowed was an alignment routine that served no purpose after liftoff on the Ariane 5 at all; it kept running because it had on Ariane 4.

Info

Ariane 501 is the standard citation for two distinct lessons: reuse is not free (code carries the assumptions of its original environment), and redundancy does not protect against design faults (two identical computers running the same wrong program fail together).

The Patriot Missile and 0.1 Seconds

On February 25, 1991, during the Gulf War, a Patriot missile battery in Dhahran, Saudi Arabia failed to track an incoming Iraqi Scud missile. The Scud struck a US Army barracks, killing 28 soldiers.

The US General Accounting Office investigation found the cause in arithmetic: the system counted time in tenths of seconds, and 0.1 has no exact representation in binary. The 24-bit fixed-point approximation introduced a tiny error each tick. The system had been designed for short deployments, but the Dhahran battery had been running for about 100 hours, long enough for the accumulated clock error of about 0.34 seconds to shift the range gate nearly 700 meters, so the system looked for the Scud in the wrong part of the sky and dismissed it. A corrected software version arrived in Dhahran the day after the strike.

Mars Climate Orbiter: Pounds vs. Newtons

On September 23, 1999, NASA’s Mars Climate Orbiter disappeared during orbital insertion at Mars. The mishap investigation found that ground software supplied by Lockheed Martin computed thruster impulse in pound-force seconds, while the navigation software at JPL expected newton-seconds, the metric unit specified in the interface documentation. Every trajectory correction had been slightly wrong for months; the spacecraft entered the Martian atmosphere far below its intended altitude and was destroyed. The lost orbiter cost about $125 million; the wider Mars Surveyor ‘98 program (which also lost its companion Polar Lander months later) totaled roughly $328 million.

The failure is remembered as “the metric mix-up,” but the mishap board emphasized process: navigators had noticed anomalies for months without the discrepancy being escalated and resolved. The bug was a symptom; the disaster was a communication failure between institutions.

The Pentium Division Bug

Intel released the Pentium on 22 March 1993, 3.1 million transistors on 294 square millimetres, and showed it to the public at CeBIT in Hannover days later. In June 1994 Thomas Nicely, a mathematics professor in Virginia, noticed that his Pentium machine gave wrong reciprocals of large primes from about the ninth significant digit. He reported it to Intel on 24 October and to colleagues on 30 October; Intel admitted on 2 November that it knew of the fault. The cause was the lookup table of the chip’s SRT division algorithm: five of its 1,066 entries had been left at zero instead of 2. Intel first played down how many users would be affected, IBM contradicted it on 12 December, and on 20 December Intel agreed to replace every faulty chip. In January 1995 it booked a charge of $475 million for the recall. The bug was in a table of constants rather than in code, but the Munich computer scientist Thomas Huckle still ranks it second on his long list of software failures.

A smaller fault showed how such errors had moved into everyday machines. In April 2015 the US Federal Aviation Administration warned that a counter in the Boeing 787’s power-control units overflowed after 248 days of continuous operation and could shut down all of the aircraft’s generators at once. Its interim remedy was the help-desk classic: switch the system off and on again at regular intervals.

Knight Capital’s 45 Minutes

On August 1, 2012, Knight Capital Group (then the largest trader in US equities) deployed new order-routing software to its eight production servers. The deployment reached only seven. The new code reused a configuration flag that, on the eighth server, activated Power Peg, a dormant test routine from 2003 that bought high and sold low to probe market behavior, with its order-tracking removed years earlier.

When the market opened, the eighth server fired millions of unintended orders across 154 stocks. In about 45 minutes the router sent more than 4 million orders while trying to fill 212 customer orders, and Knight lost more than $460 million by the SEC’s count (the firm’s own first estimate was $440 million), roughly four times its previous year’s profit. Before the open, the system had sent 97 automated e-mails warning of the error; nobody acted on them. The firm survived only days as an independent company before an emergency rescue and eventual acquisition. The SEC’s 2013 order, which fined Knight $12 million under the market access rule, became required reading for deployment engineering: no automated deployment verification, no kill switch, alerts that went unheeded, and reuse of a flag whose old meaning was still wired into production. See High-Frequency Trading for the surrounding ecosystem.

The Post Office Horizon Scandal: Trusting the Computer

The slowest-moving disaster on this list killed no one in an instant and lost no rocket; it destroyed lives through institutional faith in software. From 1999, the UK Post Office rolled out Horizon, an accounting system built by ICL/Fujitsu, to thousands of sub-post offices. ICL Pathway had won the contract in May 1996, originally together with a benefits-payment card that the government cancelled in May 1999; by 2002 the system ran some 17,500 post offices with 40,000 counters and was billed as Europe’s largest civilian IT system, with more than 3.5 million lines of code. Horizon contained defects that could produce phantom accounting shortfalls. The Post Office treated the computer’s figures as unimpeachable evidence and, between 1999 and 2015, prosecuted more than 900 subpostmasters for theft and false accounting. People were imprisoned, bankrupted, and ostracized; some died before being cleared.

One of the victims, Alan Bates, who ran a sub-post office in Wales, found an unexplained £1,000 shortfall early in 2001, lost his contract in 2003, and began organising other victims; Computer Weekly published their cases in May 2009, and the Justice for Subpostmasters Alliance formed the same year. A 2019 High Court ruling, won by Bates and 554 fellow claimants, who settled for £58 million, found Horizon contained “bugs, errors and defects” capable of causing the shortfalls. On April 23, 2021 the Court of Appeal quashed the convictions of 39 subpostmasters, and in 2024 the UK Parliament passed the Post Office (Horizon System) Offences Act, quashing the convictions en masse. The affair is widely described as the most extensive miscarriage of justice in British legal history. The disaster was less the bugs than the legal presumption that the machine was right and the human was lying.

What the Disasters Share

  • Reuse without revalidation: Therac-25 (Therac-20 code), Ariane 5 (Ariane 4 code), Knight Capital (a recycled flag). Code embeds the assumptions of its original environment, and the assumptions do not travel with it.
  • Safety margins removed because software “worked”: the Therac-25’s deleted hardware interlocks; the Post Office’s removal of human doubt.
  • Accumulation: Patriot’s rounding error and Mars Climate Orbiter’s repeated small thruster errors were each individually negligible and collectively fatal.
  • Warnings ignored: in every case operators, navigators, or auditors saw anomalies before the catastrophe; in every case the institution lacked a path for those observations to stop the system.

These cases are why memory safety, secure-by-design engineering, and disciplined deployment practice are practical concerns, and why Y2K, the disaster that didn’t happen, is best read as the industry’s one great act of preventive maintenance.

📚 Sources