Skip to content

Erlang and the Actor Model

Abstract

Erlang was developed from 1986 at the Ericsson Computer Science Laboratory in Stockholm by Joe Armstrong, Robert Virding and Mike Williams, to program telephone switches that had to run for years, survive their own faults and be upgraded while running. Its answer was a language of very cheap, isolated processes that share no memory and communicate only by messages, a practical form of the actor model, combined with supervisors that restart failed processes (“let it crash”) and the ability to replace code in a running system. Erlang began as an interpreter written in Prolog and ran about 70 times faster once Williams wrote a virtual machine in C. Ericsson banned it for new products in 1998, released it as open source the same year and saw most of its creators leave. Outside Ericsson it ran WhatsApp, RabbitMQ and, through its descendant Elixir, Discord. The often-quoted “nine nines” of availability came from one customer’s measurement on one product and is not a general property of the language.

Virding Armstrong 2013
Robert Virding (left) and Joe Armstrong, two of Erlang’s creators, in 2013. Image: Robert Scoble’s Building 43, CC BY 3.0, via Wikimedia Commons.

The Telephone Problem

Ericsson AXE Exchange
An Ericsson AXE digital telephone exchange of the 1980s. The AXE, programmed in Ericsson’s own language PLEX, earned much of the company’s profit when Erlang was being designed. Image: Televerkets fotokontor, CC0, via Wikimedia Commons.

In the mid-1980s Ericsson’s main product was the AXE telephone exchange, first produced in 1974 and developed jointly with the Swedish telecoms administration at their shared company Ellemtel. The AXE was programmed in PLEX, an Ericsson language with processes implemented partly in hardware, and by the mid-1980s it generated a large proportion of Ericsson’s profits.

Telephone software had requirements unlike most programs of the time. An exchange handled very large numbers of simultaneous calls, each a small independent activity; it had to run continuously for years; it had to keep working when hardware failed or software misbehaved; and it had to be upgraded without being switched off. Armstrong listed these requirements, from the Lab’s own analysis, as including “continuous operation over several years”, “software maintenance (reconfiguration, etc.) without stopping the system” and “fault tolerance both to hardware failures and software errors”.

The Ericsson Computer Science Laboratory, headed by Bjarne Däcker, was set up to find better ways of programming such systems. Its method was to write the same small program, a model of a plain old telephone service (“POTS”), in many languages and compare the results. A project called SPOTS, renamed DOTS and then LOTS (“Lots of DOTS”), tried Ada, Concurrent Euclid, PFL, LPL0, Frames and CLU on a VAX 11/750 running Unix. Armstrong, who had joined the Lab, programmed POTS first in Smalltalk and then in Prolog, interpreting Däcker’s instruction to “solve all of Ericsson’s software problems” generously.

From Prolog to a Language

Armstrong’s Prolog experiments turned into a meta-interpreter to which he kept adding concurrency. What started, in his words, as an experiment in “adding concurrency to Prolog” became a language of its own, and it acquired the name Erlang, probably coined by Däcker. Some said it stood for “Ericsson Language”; others said it honored the Danish mathematician Agner Krarup Erlang (1878–1929), whose work on telephone traffic founded queueing theory and gave telecom engineers the unit “erlang”. The team, Armstrong wrote, “deliberately encouraged this ambiguity.”

Robert Virding joined the implementation and rewrote much of Armstrong’s interpreter, and Mike Williams, who had worked on concurrent systems for years, taught Armstrong that three numbers mattered most for a concurrent system: the time to create a process, the time to switch between processes and the time to copy a message. Erlang processes had to be cheap in all three.

The design decisions of 1986 to 1988 went against the fashion of the time. Processes shared nothing; data sent between them was copied. Threads with shared memory protected by semaphores were the usual approach, but Armstrong and Virding argued that sharing made error recovery impossible: if a process crashed while holding shared data, the data might be left half-changed. At conferences on distributed programming, Armstrong recalled, he and Virding would take turns asking “What happens if one of the nodes crashes?”, and the usual answer was that the system would not work. Erlang’s error handling was designed around that question: processes could be linked, so that when one died, the others received a signal and could act on it.

A change log in the Prolog implementation recorded the language’s development from 24 March to 14 December 1988. “Like a child,” Armstrong wrote, “Erlang took about nine months to develop.”

Making It Fast

The first users were a group at Bollmora building a prototype system called ACS/Dunder, which used Erlang in 1988 and 1989. Their December 1989 report was positive about the language and clear about its speed: the Prolog implementation was far too slow for products, and Ericsson would need an implementation at least 40 times faster.

Experiments with compiling Erlang to concurrent logic languages such as Strand and Parlog led nowhere. Armstrong then designed a virtual machine, the JAM, and wrote a compiler for it in Prolog; Williams wrote the emulator for it in C. After tuning it ran about 70 times faster than the Prolog version, past the target of 40. Then the Bollmora group sent word that they had miscalculated: they needed 280 times.

In 1993 Bogumil Hausman, who had come to the Lab from the Swedish Institute of Computer Science, built “Turbo Erlang”, renamed BEAM (“Bogdan’s Erlang Abstract Machine”) for legal reasons. It compiled Erlang either to C, which ran about ten times faster than JAM, or to a threaded-code interpreter, which ran more than three times faster and was eventually adopted. BEAM became the standard Erlang virtual machine and gave its name to the whole runtime. Claes “Klacke” Wikström, who joined in 1990, added distribution, so that processes on different machines could exchange messages as if they were local. The book Concurrent Programming in Erlang was published in May 1993.

The Actor Model in Practice

Carl Hewitt had described the actor model in 1973: a system of independent actors that communicate only by sending messages, each able to create new actors and decide how to respond to the next message (see Concurrency and Parallelism). The Erlang team arrived at a similar design from telephony rather than from Hewitt’s theory, and Erlang became the most widely used practical implementation of it.

  • Lightweight processes. Erlang processes belong to the virtual machine, not the operating system. Each has its own small heap and is created in microseconds, and a single machine can run millions of them.
  • No shared state. Processes share no memory, so there are no locks to forget and no data races. Each process also has its own garbage collection, so collecting one process’s memory does not stop the others.
  • Asynchronous messages. A process sends a message to another process’s mailbox and continues; the receiver takes messages out with pattern matching, in whatever order it chooses.
  • Location transparency. Sending a message to a process on another machine looks the same as sending one locally.

When multicore processors arrived, the design paid off without changes to programs. The OTP team released a multiprocessor version of the runtime in August 2006. “In most other programming communities, the challenge of the multicore CPU is to answer the question, ‘How can I parallelize my program?’” Armstrong wrote. “Erlang programmers do not ask such questions; their programs are already parallel.”

Let It Crash

Erlang’s most quoted principle is “let it crash”. Instead of checking for every possible error inside a process and trying to repair its state, an Erlang process that meets an unexpected condition stops. Another process, a supervisor, notices and restarts it from a known good state. Supervisors watch other supervisors, forming supervision trees; if restarting a process does not help, its supervisor gives up and lets its own supervisor restart a larger part of the system.

Armstrong developed the reasoning in his 2003 doctoral thesis, Making reliable distributed systems in the presence of software errors: errors cannot all be prevented, so a system must be built to contain them. Combined with hot code loading, in which a new version of a module is loaded into a running system and processes switch to it at their next call, this let systems be repaired and upgraded without being stopped.

OTP and the AXD301

The Lab had moved to Ellemtel in 1988 to support AXE-N, a planned successor to the AXE that, in Armstrong’s words, “never really worked out.” Erlang’s large product came instead with the AXD301 ATM switch, announced in March 1998. Ericsson set up a product unit called OTP (Open Telecom Platform) to support the language, and OTP gave its name to the libraries that came with it: generic “behaviours” for servers, state machines, event handlers and supervisors that encode the common patterns of fault-tolerant programs. “Erlang/OTP” became the usual name for the platform. The AXD301 contained a couple of million lines of Erlang along with large amounts of C, and was probably the largest program written in a functional language at the time; Ericsson reported a fourfold increase in programmer productivity on it.

Myth: Erlang systems achieve “nine nines” (99.9999999%) availability

The figure, equivalent to about 31 milliseconds of downtime a year, comes from one customer’s measurement on one product. British Telecom reported nine nines of service availability for the AXD301 integrated over about five node-years of operation on its network. Mats Cronqvist, who worked on the AXD301 as system architect, later called the general claim “pretty bogus”: the system contained much more C than Erlang, it had no restarts or upgrades in that period, and its functionality was very well defined. Armstrong was not on the AXD301 team. The switch was very reliable, but the number describes one deployment, not a property of the language. See Myths and Misconceptions.

Banned and Released

In February 1998 Ericsson Radio Systems banned Erlang for new product development. The reason given was that “use of a proprietary language implies a continued effort to maintain and further develop the support and the development environment,” and that the company could not benefit from the wider industry’s work on mainstream languages. Armstrong later called the ban “the second most significant event in the history of Erlang.”

The team persuaded Ericsson to let the language go. After a patent review, Open Source Erlang was announced on 2 December 1998. Shortly afterwards most of the original developers, Armstrong among them, resigned and founded a company called Bluetail to build products in Erlang; Bluetail was bought by Alteon WebSystems, which was in turn bought by Nortel Networks, bringing Erlang into Nortel. “In retrospect,” Armstrong wrote, “the Erlang ban had the opposite effect and stimulated the long-term growth of Erlang.” Ericsson itself went on using Erlang, and the OTP team at Ericsson still maintains the language. Since 2001 the Uppsala University HiPE native-code compiler has been part of the distribution.

Erlang Outside Ericsson

Jose Valim 2014
José Valim, creator of Elixir, at ElixirConf in July 2014, two months before Elixir 1.0. Image: Augie De Blieck, CC BY 2.0, via Wikimedia Commons.

WhatsApp was the best-known Erlang user. Its servers were written in Erlang on FreeBSD, and in 2012 its engineers reported holding more than two million TCP connections on a single server. When Facebook bought WhatsApp in 2014 (see the WhatsApp acquisition), it had 32 engineers and about 450 million active users; in 2015 Wired reported about 50 engineers for 900 million users. The message broker RabbitMQ, the database CouchDB, the XMPP server ejabberd and the distributed database Riak were also written in Erlang.

Elixir, created by José Valim at the Brazilian consultancy Plataformatec in 2012 and released as version 1.0 on 18 September 2014, runs on the BEAM with a syntax inspired by Ruby, modern tooling and macros. Its Phoenix web framework brought Erlang’s process model to web developers. Discord built its real-time chat system in Elixir; in July 2017 it reported handling nearly five million concurrent users. Gleam, a statically typed language for the BEAM, reached version 1.0 on 4 March 2024. The actor model also reappeared on other platforms, in Akka for the JVM and Microsoft’s Orleans.

Joe Armstrong died on 20 April 2019, at 68.

Dead End

Erlang’s own history contains the dead end that nearly ended it: the attempt to make it fast by compiling it to concurrent logic programming languages. In the late 1980s parallel Prolog derivatives such as Parlog, KL1 (the language of Japan’s Fifth Generation Computer project) and Strand were where much research on concurrent languages was going. Erlang had grown out of Prolog, so studying how those languages were implemented seemed natural. “This actually turned out to be a mistake,” Armstrong wrote. The compilation to Strand worked, but the logic-language machinery did not fit Erlang’s semantics, and the team, as Armstrong put it, “were stuck. We knew what we wanted to do but not how to do it.” The way out was to abandon the logic-programming heritage for implementation purposes and build a conventional abstract machine and emulator in C, the JAM and later the BEAM. The concurrent logic languages themselves faded with the Fifth Generation project in the early 1990s, while Erlang’s simpler process model survived.

📚 Sources