Skip to content

Brewster Kahle and the Internet Archive

Abstract

The web forgets by default: the average page lives on the order of months before it moves or vanishes. Brewster Kahle (b. 1960) treated that as an emergency and built the institution to fix it: the Internet Archive, founded in San Francisco in May 1996 with the motto “universal access to all knowledge.” Its Wayback Machine (named for the cartoon time machine in Rocky and Bullwinkle) opened to the public in October 2001 with more than 10 billion pages and archived its trillionth page in October 2025, alongside millions of books, films, TV news broadcasts, sound recordings, and runnable vintage software. Kahle funded it the Valley way, selling two search startups to AOL and Amazon, then spent the fortune building a library. In the 2020s the library was sued by book publishers and record labels, hacked, and knocked offline for weeks, while the rest of the web’s memory (GeoCities, Google’s cache, entire federal agency websites) was switched off around it. In 2026 the Computer History Museum made Kahle a Fellow.

Brewster Kahle
Brewster Kahle, 2009. Image: Joi Ito, CC BY 2.0, via Wikimedia Commons.

From the Connection Machine to WAIS

Connection Machine CM-2
Connection Machine CM-2 with its DataVault storage unit, Computer Museum of America. Image: Logg Tandy, CC BY-SA 4.0, via Wikimedia Commons.

Kahle was born on October 21, 1960, in New York City. He studied computer science and engineering at MIT (B.S. 1982) under Marvin Minsky and Danny Hillis, then followed Hillis to Thinking Machines (1983–1989) as a lead engineer on the Connection Machine, the massively parallel supercomputer of the AI-hardware era. Thinking Machines needed applications that justified 65,536 processors, and searching large text collections was one of them. Kahle took the idea out of the lab.

At Thinking Machines he led the development of WAIS (Wide Area Information Servers), the first distributed search-and-retrieval system on the internet, built from 1990 as a joint project of Thinking Machines, Apple, Dow Jones and the accounting firm KPMG Peat Marwick, and its protocol borrowed from Z39.50, the library-catalog standard. A user typed a question in plain words; WAIS queried servers across the network and ranked the documents that came back. It was a pre-web rival to Gopher and HTTP for how networked documents would be found (see Dead End: Gopher for how that contest ended). Kahle and Harry Morris left Thinking Machines to found WAIS Inc. in Menlo Park in 1992, which sold to AOL in May 1995 for $15 million.

Alexa and the Crawl

In 1996 Kahle and Bruce Gilliat founded Alexa Internet, which crawled the web to rank sites and recommend related ones through a browser toolbar. The same year, in May 1996, Kahle founded the Internet Archive as a nonprofit. The pairing was deliberate: the company crawled, and the nonprofit received the crawls and kept them. Alexa sold to Amazon in 1999 for $250 million in stock. Kahle kept the crawls.

For the first five years the archive was invisible to the public. The data went onto digital tape, and the only way to see an old page was to be one of the few researchers who knew how to ask.

A Library for the Web

Internet Archive Headquarters
The Internet Archive’s headquarters on Funston Avenue, San Francisco, a former Christian Science church. Image: Svobodat, CC BY-SA 3.0, via Wikimedia Commons.

In October 2001 the Wayback Machine opened the collection: type a URL, see the page as it was on a chosen date. It went live with more than 10 billion archived pages. A mirror went to the Bibliotheca Alexandrina in Egypt, so the new Library of Alexandria holds a backup of the new library of everything, a redundancy the old Alexandria never had.

In 2009 the Archive moved into a former Christian Science church on Funston Avenue in San Francisco. Its Greek-revival columns look like what the building now is, a library.

The collection grew into the web’s memory and then past it. By September 2024 it held 866+ billion web pages, 42.5 million texts, 13 million videos, 3 million TV news broadcasts, 14 million audio recordings, 1.2 million software titles. Two features changed how people used it. Save Page Now (October 2013) let anyone archive a page on demand, which turned the Wayback Machine into a citation tool for journalists and Wikipedia editors. In April 2017 the Archive stopped honoring robots.txt exclusions retroactively and began handling removal on request, after too many domain squatters had used a new robots.txt to erase a dead site’s entire history.

The Petabox

Internet Archive Petabox
A Petabox rack of Internet Archive storage servers. Image: roens, CC BY-SA 2.0, via Wikimedia Commons.

The Archive does not rent cloud storage. It designed its own storage racks, the Petabox, dense shelves of commodity hard drives built for low power and cheap maintenance, and it runs them in its own buildings. The totals show the curve: about 3 petabytes in 2009, 25 petabytes in September 2018, more than 70 petabytes in December 2020, and about 100 petabytes by October 2025, not counting copies.

Copies are the point. After the US election of November 2016, Kahle announced that the Archive would build a copy of its collections in Canada. The Archive had been planning a partial backup there already; the election, Kahle wrote, “ramped us into higher gear.” At the time the collection was about 26 petabytes and the copy was expected to cost millions of dollars. The Wayback Machine did not move. It duplicated itself across a border.

Books, Records and Software

The web was the first collection, not the only one.

  • Books. In 2005 Kahle organized the Open Content Alliance with libraries and technology companies as an open alternative to Google’s book-scanning project, which kept its scans inside Google’s own services. The Archive built camera-based book scanners and ran them in dozens of scanning centers, many inside partner libraries. Open Library, a project to give every book ever published a web page, was built with Aaron Swartz (see Aaron Swartz). From 2011 the Archive also kept the physical originals: climate-controlled shipping containers in Richmond, California, held about 500,000 volumes in the first years, on the librarian’s principle that you keep the original after you copy it. In 2024 Leiden University Library sent it 350,000 European dissertations from 1850 to 1990.
  • Fire. In the early hours of November 6, 2013 (the fire brigade was called out at about 3:45 a.m.), a side building at the headquarters that housed one of the Archive’s scanning centers burned. Kahle put the damage at around $600,000: cameras, lights, scanning equipment and about 20 boxes of books and film, most already digitized. No one was hurt and no data was lost.
  • Records. The Great 78 Project digitized shellac 78-rpm records, recordings from roughly 1880 to 1960, including the hiss and crackle of the specific worn copy, which the Archive argued was part of the historical record. By 2024 it had digitized more than 400,000 recordings.
  • Television. In September 2012 the Archive opened a searchable TV News Archive of US news broadcasts, searchable through their closed captions.
  • Software. From 2013 to 2014 the Archive began running old software in the browser via emulation: arcade games, MS-DOS games and vintage programs, playable without installing anything. It made the Archive a working museum of digital culture.

The lineage is explicit: Vannevar Bush’s Memex, Ted Nelson’s Xanadu, and Wikipedia are the Archive’s intellectual siblings.

In Court Before the Siege

Kahle took the Archive’s legal position on the offensive early. In 2004 he, the Archive, and the film archivist Rick Prelinger sued the US Attorney General (Kahle v. Ashcroft, later Kahle v. Gonzales), represented by Lawrence Lessig. The argument was about orphan works: US copyright once had to be renewed after its first term, and most works nobody cared about fell into the public domain. The Copyright Renewal Act of 1992 made renewal automatic, which locked up millions of works whose owners could no longer be found. The Ninth Circuit rejected the challenge on January 22, 2007, saying it was “essentially the same argument, in different form, that the Supreme Court rejected in Eldred.” The Supreme Court declined to hear the case on January 7, 2008.

The Archive did better against the FBI. Served with a National Security Letter demanding user records, and a gag order forbidding it to say so, Kahle fought it with the EFF and ACLU. In 2008 the FBI withdrew the letter and lifted the gag, one of the very few NSLs ever beaten in the open (see The Privacy War).

The Siege Years

The 2020s tested whether a library for the digital age is legal in it.

  • Books. The Archive lends scanned books under controlled digital lending: one physical copy owned, one digital copy loaned at a time. On March 24, 2020, as COVID closed libraries, it dropped the waitlists and called the result the National Emergency Library. Hachette, HarperCollins, Wiley and Penguin Random House sued on June 1, 2020, over 127 titles; the Archive ended the emergency library on June 16. Judge John Koeltl ruled for the publishers on March 24, 2023. The consent judgment that August barred lending of the plaintiffs’ books where a commercial e-book exists, and more than 500,000 books were eventually taken out of lending. The Second Circuit affirmed on September 4, 2024, and the Archive let the December 2024 deadline for a Supreme Court petition pass. The core theory of controlled digital lending, that a library may lend digitally what it owns physically, had lost (see Copyright and IP in the Digital Age).
  • Music. In August 2023 record labels led by UMG and Sony sued over the Great 78 Project, seeking statutory damages that grew from about $400 million in 2024 to $621 million in the amended complaint of 2025, an existential number for a nonprofit with revenue of $26.8 million (2024). The labels argued there was “zero basis” for claiming that “the hisses, crackles, and pops” gave anyone the right to copy recordings on a mass scale. The parties reached a confidential settlement in September 2025, and the case was dismissed on October 16, 2025.
  • Attack. In September 2024 attackers took a user database with about 31 million records. On October 9, 2024, a distributed denial-of-service attack took the site down; the Wayback Machine came back in read-only mode on October 14, and new captures resumed only on November 4. For a month the web’s memory was one under-funded nonprofit with a single headquarters, and it was offline.

Kahle’s framing: every earlier medium’s history survived because libraries bought, kept and lent copies without asking permission; digital licensing replaces ownership with rental, and a library that can only rent cannot preserve. The publishers’ framing: scanning and lending entire in-print books is infringement at scale, whatever the mission. The courts agreed with the publishers about books. The publishers’ lawyers kept citing the Wayback Machine.

The Record of Record

The Archive’s evidentiary role became routine. The US and European patent offices accept Wayback timestamps as evidence of prior art; litigants have offered archived pages in US courts since at least 2004, when a Chicago federal judge still refused to admit them; journalists cite them as the standard proof of “what the web said.”

In September 2024 Google, which had retired its own cache of web pages earlier that year, began linking search results to the Wayback Machine. In the End of Term crawl of 2024/25, run by the Archive with Harvard’s Library Innovation Lab, the Stanford and University of North Texas libraries, Webrecorder, Common Crawl and the volunteer Archive Team, the harvest of US federal websites grew to 2.29 petabytes, against 266 terabytes four years earlier. It became unusually useful: from January 2025, large amounts of material vanished from federal websites, and some agencies, such as USAID, disappeared from the web entirely. The copies were in the Wayback Machine.

On July 24, 2025, the Archive was designated a US federal depository library. On October 22, 2025, it celebrated its trillionth archived web page.

Kahle collected the recognition that goes with a public institution: fellow of the American Academy of Arts and Sciences (2005), member of the National Academy of Engineering (2010), Internet Hall of Fame (2012), and in April 2026 a Computer History Museum Fellow Award, alongside Jeff Hawkins, Donna Dubinsky and Ed Colligan of Palm and the computer-music pioneer John Chowning. He and his wife, Mary Austin, also give through the Kahle/Austin Foundation, whose grants have included the Free Software Foundation’s GNU Project (see Richard Stallman and GNU).

⚠️ Dead End: The Web That Wasn’t Saved

The Archive’s importance shows most clearly where it wasn’t. GeoCities (38 million pages of the amateur 1990s web) was deleted by Yahoo! in 2009 with months of notice; only scramble efforts by the Archive and the volunteer Archive Team saved a fraction (see Dead End: Yahoo!). Google Cache, the de facto second copy of the live web, was retired in 2024. Link rot is the norm: studies of US Supreme Court opinions found roughly half their cited links dead, and by 2021 a quarter of all deep links in New York Times articles published since 1996 no longer worked (see Link Rot and the Digital Dark Age). Without the crawlers, the primary sources of the internet era (the pages, the propaganda, the retracted claims, the deleted posts of the powerful, the vanished government reports) would be gone. Earlier chapters of this encyclopedia rest on many institutions that kept the paper. For the web there is essentially one, funded by donations, sued for its book and record collections, and hacked once already.

Fun Fact

The Wayback Machine is named after the WABAC Machine from the cartoon The Rocky and Bullwinkle Show, in which a boy and his time-traveling dog visit history. Inside the Archive’s church headquarters stand rows of three-foot ceramic statues: every employee who has worked there three years gets one, a terracotta army of librarians.

See also: Alexandra Elbakyan and Sci-Hub.

📚 Sources