Skip to content

Ivan Sutherland and Sketchpad: The First Graphical Interface

Abstract

In January 1963, Ivan Sutherland submitted his MIT PhD thesis on Sketchpad, a program he had written for the TX-2 computer at Lincoln Laboratory. It allowed a user to draw geometric figures on a cathode-ray tube display using a light pen, manipulate them interactively, define constraints between parts, and copy and instantiate them as reusable components. Its master drawings and instances anticipated object-oriented programming, and Alan Kay named the thesis as one of the two starting points of Smalltalk. Sutherland was twenty-four when he submitted it. In 1968 he built one of the first computer-driven head-mounted displays and co-founded Evans & Sutherland, whose hardware put the geometry pipeline of 3D graphics into silicon. He received the Turing Award in 1988.

Ivan Sutherland
Ivan Sutherland at Babbage Difference Engine No. 2. Image: Richard F. Lyon, CC BY-SA 3.0, via Wikimedia Commons.

The TX-2 and the Light Pen

Ivan Edward Sutherland was born on 16 May 1938 in Hastings, Nebraska, the son of a civil engineer, and grew up partly in Scarsdale, New York. At twelve he was working with Edmund Berkeley’s relay computer Simon, and for a school project he built a 256-bit drum memory. As an engineering student at Carnegie Tech in Pittsburgh he built Machina Versatilis in 1957, the first transistorised electronic “tortoise” in the cybernetic tradition, and after a year of graduate study in California he went east. From 1961 he worked on his doctorate at MIT, with Claude Shannon as his adviser. The machine he had access to was Lincoln Laboratory’s TX-2, an experimental transistor-based computer designed by Wesley Clark and operational since 1958, with an unusual feature for its era: a cathode-ray tube display that could show arbitrary graphics, not just text. It came with a light pen, a device that could detect the position on the screen where it was pointed by sensing the electron beam as it refreshed the phosphor.

The combination (a display that could show graphics, and an input device that could point to positions on that display) was the raw material Sutherland needed. The question was what to do with it.

His answer was Sketchpad: a program that would let a user draw on the computer screen in real time and manipulate what they drew through a combination of light pen gestures and keyboard commands.

What Sketchpad Did

Sketchpad’s capabilities included features that did not become standard in commercial software for twenty years:

Interactive drawing: The user drew lines and curves by pointing with the light pen. Lines snapped to geometric precision, a capability the user could invoke to turn an approximate gesture into an exact segment. The display updated instantly as the user worked.

Rubber-band lines: When drawing a line, the endpoint under construction followed the light pen position in real time, stretching like a rubber band before being fixed. This technique (now universal in every drawing program) appeared here for the first time.

Constraints: Sketchpad allowed users to declare geometric relationships between elements (“this line is horizontal,” “these two lines are perpendicular,” “this circle is tangent to that line”) and the system would maintain those constraints as the user manipulated other parts of the drawing. If you moved a constrained element, the system solved the constraint equations and adjusted everything else accordingly. This was the first constraint solver in a graphical interface.

Master drawings and instances: A user could define a drawing as a “master” and then place multiple instances of it elsewhere in the drawing. Modifying the master updated all instances. This is object-oriented programming before object-oriented programming had a name: a class and its instances, with inheritance.

Hierarchical structure: Drawings could contain sub-drawings, which could contain further sub-drawings, creating a tree of components. The system maintained the hierarchy and applied transformations correctly at each level.

Zooming: The user could magnify any portion of the drawing to work in detail, then zoom out. The drawing existed at arbitrary precision, not limited to the pixel grid of the display.

Copy and delete: elements could be duplicated and erased.

The Constraint Solver

Sketchpad’s constraint system was technically the most sophisticated part of the program. Maintaining geometric constraints (ensuring that specified relationships between drawing elements remain true as the user modifies the drawing) requires solving systems of equations in real time. Sutherland used two methods: a one-pass method that ordered the constraints and satisfied them in sequence where that was possible, and numerical relaxation, an iterative approximation, as the fallback. Constraint solving is now used in UI layout (iOS’s Auto Layout, Android’s ConstraintLayout), CAD software and 3D modeling tools.

The Object-Oriented Insight

The most intellectually consequential aspect of Sketchpad was not a user interface feature but a data structure: the way Sutherland organized the program’s internal representation of drawings.

Each element in a Sketchpad drawing was a data block in a ring structure, and each type of element carried pointers to the subroutines that handled it, an early form of what object-oriented languages later called virtual functions. Master drawings were templates from which instances were created; each instance took the master’s structure but kept its own position, scale and rotation.

Templates, instances and type-specific behaviour are the core of object-oriented programming. Sutherland reached them from the needs of a graphical system, before any object-oriented programming language existed.

Alan Kay arrived at Utah as a graduate student in 1966 and, by his own account in “The Early History of Smalltalk”, found two things waiting on his desk: a copy of Sutherland’s thesis and a tape of the Simula language. He counted both as the origins of Smalltalk, the language that named object-oriented programming and fed into C++, Java, Python, Ruby and Swift.

Head-Mounted Display and Virtual Reality

Sutherland completed his PhD in 1963, served in the Army as a first lieutenant, and moved to ARPA, where he succeeded J.C.R. Licklider as director of the Information Processing Techniques Office (IPTO) from 1964. He was twenty-six when he took the role.

At the end of 1965 he moved to Harvard as an associate professor. He built on earlier work, from stereoscopy and Morton Heilig’s experiments to Philco’s camera-based Headsight of 1961, but was the first to put a computer and computer graphics behind the display. In 1968, with his student Bob Sproull and others, he built a head-mounted display that placed small screens in front of each eye, displaying a stereoscopic computer-generated image that changed as the user moved their head. The system was so heavy it had to be suspended from the ceiling by a mechanical arm; its creators called it the Sword of Damocles. The image it displayed was a simple wireframe room, a few lines representing the walls, floor, and ceiling of a virtual space that the user appeared to inhabit.

The Sword of Damocles is usually counted as the first virtual reality system. It combined stereoscopic display, head tracking and a computer-generated environment, the elements VR headsets still use. Sutherland described it in the 1968 paper “A Head-Mounted Three-Dimensional Display.”

“The Ultimate Display” (1965)

Three years before the Sword of Damocles, Sutherland published a paper called “The Ultimate Display” that described what a fully realized virtual reality system might eventually achieve: a room within which a computer could control the existence of matter, where a chair displayed in the room would be good enough to sit in and a bullet displayed in the room would be good enough to kill you. The paper fills three pages of the congress proceedings and is still cited in virtual and augmented reality research.

Evans & Sutherland: Building the GPU’s Ancestor

In 1968, Sutherland co-founded Evans & Sutherland with David Evans in Salt Lake City, next to the University of Utah where both taught. The company built specialized hardware for real-time 3D graphics and flight simulators.

The E&S LDS-1 (Line Drawing System, 1969) and the later Picture System were vector displays that performed coordinate transformation and clipping in dedicated hardware stages, one after another, so that a picture could be moved and rotated in real time. Sutherland’s work on clipping (the Cohen–Sutherland algorithm with Danny Cohen in 1967, and the Sproull–Sutherland clipping divider) fed into this design. The same sequence of stages, extended by rasterization and shading, became the geometry pipeline of later graphics hardware, including the consumer GPUs of the 1990s.

E&S flight simulators trained commercial and military pilots for decades.

The University of Utah’s computer graphics program, which Sutherland helped shape, produced an extraordinary concentration of foundational contributors: Jim Clark (Silicon Graphics, Netscape), John Warnock (Adobe, PostScript), Alan Kay (Smalltalk, Xerox PARC), Ed Catmull (Pixar, computer animation), Henri Gouraud (Gouraud shading), Bui Tuong Phong (Phong shading and reflection model).

The Turing Award

Sutherland taught at Utah from 1968 to 1974 and at Caltech from 1974 to 1978, where he was the founding head of the computer science department. His consulting firm Sutherland, Sproull and Associates was bought by Sun Microsystems in 1990 and became the seed of Sun Labs. Since 2009 he has led asynchronous-circuit research at Portland State University in Oregon.

Sutherland received the ACM Turing Award in 1988, citation:

“For his pioneering and visionary contributions to computer graphics, starting with Sketchpad, and continuing after.”

Alan Kay tells of asking Sutherland how he had managed the first interactive graphics program, the first non-procedural programming language and the first object-oriented software system all in one year. The answer: “Well, I didn’t know it was hard.”

Dead End: Sketchpad’s Premature Power

Sketchpad’s most important limitation was the machine it ran on. The TX-2 was a one-of-a-kind experimental computer at Lincoln Laboratory, not a commercial product. The program could not run anywhere else. Its capabilities (interactive graphics, constraint solving, hierarchical object structures) required hardware that commercial computers would not provide for nearly two decades.

The interactive computing revolution that Licklider had envisioned and that Sketchpad demonstrated had to wait for hardware to catch up. When Doug Engelbart demonstrated the NLS system in 1968 (with mouse, windows, and collaborative editing) he too used one-off research hardware. Interactive graphics of Sketchpad’s kind reached commercial users only with CAD workstations and, in the early 1980s, personal computers with graphical displays.

This meant that Sketchpad’s direct influence on commercial computing was mediated through researchers who had seen the demonstration or read the thesis, rather than through a product that users could experience directly. The ideas survived and propagated; the specific system was a historical artifact for most of the period between its demonstration and the arrival of personal computers.

📚 Sources