Skip to content

Claude Shannon and Information Theory

Abstract

In 1948, a thirty-two-year-old mathematician at Bell Labs published a two-part, 79-page paper that founded a new science. Claude Shannon’s “A Mathematical Theory of Communication” answered a question nobody had formally posed: what is information, stripped of meaning, context, and intention? His answer: that information could be measured in bits, that every channel has a maximum capacity, and that noise could always be overcome with the right encoding, gave engineers a map of the possible. Every digital system built since, from satellite links to mobile phones to streaming video, operates within bounds that Shannon proved.

Claude Shannon
Claude Shannon. Image: Tekniska Museet of Sweden, CC BY 2.0, via Wikimedia Commons.

The Restless Mind at Bell Labs

Claude Elwood Shannon was born on April 30, 1916, in Petoskey, Michigan, a small resort town on the shore of Lake Michigan. His father was a businessman and for a time a probate judge; his mother was a language teacher and principal of the high school in Gaylord, where he grew up. Shannon showed an early gift for electrical tinkering (he built a telegraph system connected by barbed wire between his house and a friend’s a half-mile away) and studied electrical engineering and mathematics simultaneously at the University of Michigan, graduating in 1936. He went to MIT for graduate work, and there, at the age of twenty-one, he produced what is arguably the most important master’s thesis in the history of technology.

Shannon’s 1937 master’s thesis applied the Boolean algebra of George Boole (a nineteenth-century logical formalism originally developed to analyze syllogisms) to the design of electrical relay circuits. The key insight was that a relay, which is either open or closed, corresponds perfectly to a Boolean variable, which is either false or true. Complex logical operations could therefore be implemented in electrical circuits; conversely, any circuit could be analyzed using Boolean algebra. The thesis provided the theoretical foundation for digital circuit design, the basis of every computer, smartphone, and digital device ever built. Shannon was twenty-one.

He completed a doctorate in mathematics at MIT in 1940 with a thesis on theoretical genetics, “An Algebra for Theoretical Genetics,” spent a year as a National Research Fellow at the Institute for Advanced Study, and in 1941 joined Bell Telephone Laboratories in New York, the great industrial research institution that also employed William Shockley, John Bardeen, and Walter Brattain. Bell Labs was, in the 1940s, perhaps the greatest concentration of scientific talent ever assembled in a corporate setting. Shannon fit right in.

The Problem He Was Born to Solve

Before Shannon, communication engineering was an empirical art. Engineers designed telephone circuits, radio transmitters, and telegraph systems through experience, intuition, and trial and error. They knew that noise degraded signals. They knew that bandwidth mattered. They knew that compression was sometimes possible. But they had no mathematical framework that unified these intuitions, no way to determine whether a given system was near its theoretical limit or could be dramatically improved, no answer to the question of whether error-free communication over a noisy channel was even possible in principle.

Shannon’s central insight was both simple and revolutionary: separate information from meaning. The semantic content of a message (whether it was love poetry or stock prices, profound or trivial) was irrelevant to the engineering problem of transmitting it reliably from one place to another. What mattered was surprise: how much did receiving this message reduce the recipient’s uncertainty about the world? Information, in Shannon’s framework, was a measure of unpredictability.

This move (treating information as a mathematical quantity divorced from meaning) was philosophically bold and practically liberating. It meant that the same theory could apply to telephones, radio, optical fibers, and media not yet invented. The theory was about the channel, not the content.

The Bit and the Entropy Formula

Shannon defined the bit (binary digit) as the fundamental unit of information: the amount of uncertainty resolved by learning which of two equally likely alternatives is true. A fair coin flip conveys exactly one bit of information. If you already know the coin is biased (say, heads ninety percent of the time) a single flip conveys less than one bit, because you had less uncertainty to begin with.

For a source that produces messages drawn from an alphabet with known probabilities, Shannon derived the entropy formula:

H = −Σ p(x) log₂ p(x)

where p(x) is the probability of each symbol. Entropy measures the average uncertainty of the source. A source that always emits the same symbol has zero entropy, no information. A source with a uniform distribution over n symbols has log₂(n) bits of entropy, maximum information per symbol. The formula has the same form as the entropy of statistical mechanics in the work of Ludwig Boltzmann and J. Willard Gibbs. Shannon used the same word deliberately, reportedly on the advice of John von Neumann, who told him: “You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage.”

Info

The term “bit” was not coined by Shannon. It was coined by John Tukey, the statistician who also developed the Fast Fourier Transform, in a Bell Labs memo written shortly before Shannon’s paper appeared. Shannon adopted Tukey’s term as the natural unit for his theory. Tukey later also coined the word “software.”

The Shannon Limit

The second and more immediately engineering-relevant result of Shannon’s 1948 paper was the channel capacity theorem. For any communications channel (a copper wire, a radio link, an optical fiber, a satellite transponder) there exists a maximum rate at which information can be transmitted with arbitrarily small error probability, regardless of how much noise is present. This rate, measured in bits per second, depends only on the channel’s bandwidth and signal-to-noise ratio:

C = B log₂(1 + S/N)

This is the Shannon-Hartley theorem. Its implications were startling. First, it proved that error-free communication over a noisy channel was possible, not just in some ideal limiting case, but achievable in principle by clever enough encoding. Engineers had generally assumed that noise was a fundamental, unavoidable barrier to accuracy. Shannon showed they were wrong. Second, the theorem told you the hard ceiling on any possible system. A communication system operating at the Shannon limit cannot be improved; one operating below it is wasting capacity that better coding schemes could recover.

The practical challenge was to find codes that approached the Shannon limit. Shannon proved existence (such codes must exist) but his proof was non-constructive. Finding them took decades. Early error-correcting codes by Richard Hamming (1950) and Reed-Solomon (1960) were practical but far from the limit. Turbo codes, presented by Claude Berrou and colleagues in 1993 to an audience that at first doubted the results, came within a fraction of a decibel of the Shannon limit and went into 3G and 4G mobile networks. LDPC codes, invented by Robert Gallager in 1960 and then largely forgotten for thirty years, were rediscovered in the 1990s and now carry the data channels of 5G. The fifty-year gap between Shannon’s proof and practical codes approaching his limit is one of the great stories of applied mathematics.

The Master’s Thesis and Boolean Logic

It is worth pausing on Shannon’s 1937 thesis, which predates the information theory paper by eleven years but deserves equal recognition. Before Shannon, Boolean algebra was a mathematical curiosity, a formal system for analyzing logical propositions invented by George Boole in 1854 with no obvious practical application. After Shannon’s thesis, Boolean algebra was the mathematical language of circuit design.

Shannon showed that a two-terminal electrical network (a combination of switches, relays, and connections) corresponds exactly to a Boolean expression, with series connections representing logical AND and parallel connections representing logical OR (in today’s convention; Shannon’s own “hindrance” notation was the dual, with series as addition). Every circuit could be described algebraically; every Boolean expression could be wired. This meant that circuit designers could use algebraic manipulation to simplify circuits, that any logical function could be implemented electrically, and that digital computers (machines built from Boolean logic gates) were theoretically possible.

Shannon had come to MIT to run Vannevar Bush’s differential analyzer, whose relay control circuits set him thinking. The psychologist Howard Gardner later called the thesis “possibly the most important, and also the most famous, master’s thesis of the century.”

The Menagerie of Machines

Shannon’s intellectual personality was defined by an obsessive curiosity that ranged freely across disciplines and refused to distinguish between serious work and play. At Bell Labs, he built a chess-playing machine; he published a foundational paper on computer chess in 1950, describing two strategies (exhaustive look-ahead and selective evaluation) that remained the basis of chess programs for half a century. In 1950, with his wife Betty, he built Theseus, a maze-solving machine: a “feeler finger” under the floor of a 25-square maze, carrying a magnet that pulled a metal mouse above it, which turned 90 degrees whenever its whiskers touched a wall; the route was stored in relays, two per square. He showed it at a cybernetics meeting in New York in March 1951. The first run took about two minutes; once the route was stored, the mouse did it in fifteen seconds. It was one of the first demonstrations of machine learning, and the original box is in the MIT Museum. He built THROBAC, a calculator that performed arithmetic in Roman numerals. He built a machine whose sole purpose was to turn itself off when you switched it on, an idea of Marvin Minsky’s, and game machines for Hex and Nim. His switching-circuit ideas went into Edmund Berkeley’s GENIAC kit, and an advertisement of October 1961 presented the Minivac 601, an $85 relay “computer” for home experiments sold by the Scientific Development Corporation of Watertown, as Shannon’s private project. It was programmed by patch cords, like ENIAC, had a motor-driven sixteen-position rotary switch for short sequences, and could play Nim and tic-tac-toe; the improved Minivac 6010 followed in 1962 at $155.

He juggled constantly and rode a unicycle through the corridors of Bell Labs, sometimes simultaneously. He built juggling machines and worked out a mathematical theorem of juggling. He collected motorized pogo sticks and unicycles. He was, by all accounts, genuinely playful in a way that extended to mathematics: he pursued problems because they were interesting, not because they were prestigious.

Info

Shannon’s 1948 paper was actually a somewhat compressed version of ideas he had been developing since the early 1940s. A classified wartime report, “A Mathematical Theory of Cryptography” (1945), applied the same framework to cryptography, proving that the one-time pad was theoretically perfect and establishing the mathematical basis for information-theoretic security. The report was declassified in 1949 and published as “Communication Theory of Secrecy Systems.” Shannon thus founded both information theory and mathematical cryptography in the same decade.

The Reluctant Celebrity

Shannon was deeply uncomfortable with fame. He attended few conferences, declined most speaking invitations, and spent his later years at MIT pursuing mathematics for its own sake. In 1973 he delivered the inaugural Shannon Lecture at the International Symposium on Information Theory in Ashkelon, Israel (the field’s highest honor, named for him) and then largely vanished from the community. When he reappeared, unannounced, at the 1985 symposium in Brighton, England, the room reportedly fell silent: colleagues described it as something like seeing a figure from history appear in the flesh. Coaxed into speaking at the banquet, he talked briefly, then pulled three balls from his pockets and began to juggle.

He had moved from Bell Labs to MIT in 1956, where he held an endowed chair until his retirement. His office was legendarily cluttered with mechanical toys, half-built devices, and mathematical notes. He continued producing occasional papers of great quality: on mathematical games, on artificial intelligence, on the complexity of computing. He stopped publishing in the early 1970s, not because he had stopped thinking but because his standards for what merited publication had become impossibly high.

Shannon developed Alzheimer’s disease in the 1990s. He spent his final years in a nursing home in Medford, Massachusetts, where former colleagues occasionally visited. He died on February 24, 2001. He never became a household name; few people outside engineering and mathematics recognized his name. This obscurity is extraordinary given the scope of his influence. Every time a video streams without stuttering, every time a phone call arrives clearly despite interference, every time data is compressed or encrypted, Shannon’s theorems are silently doing their work.

The Legacy in Every Digital System

Information theory is not a historical curiosity. It is a living discipline that continues to generate new mathematics and new technology. The development of efficient codes that approach the Shannon limit remains an active research area. The extension of Shannon’s framework to quantum channels (quantum information theory, which determines what quantum computers can and cannot communicate) is one of the most active fields in physics and computer science.

Shannon’s framework also influenced fields he never intended. Molecular biology found that DNA could be analyzed as an information channel, the genetic code has a capacity, mutations are noise, and evolution is a kind of coding optimization. Neuroscience borrowed the entropy framework to analyze neural firing patterns. Linguistics borrowed the entropy measure to study the redundancy of natural language. Shannon himself warned against these extensions in his 1956 essay “The Bandwagon,” arguing that attaching the word “information” to a field did not automatically import the technical machinery of information theory. The warning was largely ignored, which is arguably a tribute to how attractive his framework was.

The connection to artificial intelligence and computation is explored further in The Rise of Artificial Intelligence. The Boolean logic foundations of digital circuits that Shannon’s master’s thesis established connect directly to the hardware revolution described in The Integrated Circuit Revolution.

📚 Sources