Ivan Sutherland and Sketchpad: The First Graphical Interface

Zusammenfassung

In January 1963, Ivan Sutherland demonstrated his MIT PhD thesis on a TX-2 computer at Lincoln Laboratory. The program was called Sketchpad. It allowed a user to draw geometric figures on a cathode-ray tube display using a light pen, manipulate them interactively, define constraints between parts, and copy and instantiate them as reusable components. Every concept it introduced — windows, object-oriented structure, hierarchical constraint systems, interactive graphical manipulation, rubber-band lines — is still in every computing system built today. Alan Kay, who built Smalltalk and the conceptual foundation of the personal computer, said Sketchpad was “the greatest single programming achievement before or after.” Sutherland received the Turing Award in 1988. He was twenty-four years old when he completed Sketchpad. He then spent the following decade inventing virtual reality, and the decade after that building the hardware architecture underlying every modern GPU. The field of computer graphics is substantially the extended footnote to one person’s PhD thesis.

The TX-2 and the Light Pen

Sutherland arrived at MIT’s Lincoln Laboratory in 1960 to work on his doctorate under Claude Shannon. The machine he had access to was the TX-2 — an experimental transistor-based computer with an unusual feature for its era: a cathode-ray tube display that could show arbitrary graphics, not just text. The TX-2 had been built to explore human-computer interaction; it came with a light pen, a device that could detect the position on the screen where it was pointed by sensing the electron beam as it refreshed the phosphor.

The combination — a display that could show graphics, and an input device that could point to positions on that display — was the raw material Sutherland needed. The question was what to do with it.

His answer was Sketchpad: a program that would let a user draw on the computer screen in real time and manipulate what they drew through a combination of light pen gestures and keyboard commands.

What Sketchpad Did

Sketchpad’s capabilities, demonstrated in 1963, included features that would not become standard in commercial software for twenty years:

Interactive drawing: The user drew lines and curves by pointing with the light pen. Lines snapped to geometric precision — a capability the user could invoke to turn an approximate gesture into an exact segment. The display updated instantly as the user worked.

Rubber-band lines: When drawing a line, the endpoint under construction followed the light pen position in real time, stretching like a rubber band before being fixed. This technique — now universal in every drawing program — appeared here for the first time.

Constraints: Sketchpad allowed users to declare geometric relationships between elements — “this line is horizontal,” “these two lines are perpendicular,” “this circle is tangent to that line” — and the system would maintain those constraints as the user manipulated other parts of the drawing. If you moved a constrained element, the system solved the constraint equations and adjusted everything else accordingly. This was the first constraint solver in a graphical interface.

Master drawings and instances: A user could define a drawing as a “master” and then place multiple instances of it elsewhere in the drawing. Modifying the master updated all instances. This is object-oriented programming before object-oriented programming had a name: a class and its instances, with inheritance.

Hierarchical structure: Drawings could contain sub-drawings, which could contain further sub-drawings, creating a tree of components. The system maintained the hierarchy and applied transformations correctly at each level.

Zooming: The user could magnify any portion of the drawing to work in detail, then zoom out. The drawing existed at arbitrary precision, not limited to the pixel grid of the display.

Copy and undo: Sketchpad supported operations that are now fundamental: duplicating elements, and undoing recent actions.

The Constraint Solver

Sketchpad’s constraint system was technically the most sophisticated part of the program. Maintaining geometric constraints — ensuring that specified relationships between drawing elements remain true as the user modifies the drawing — requires solving systems of equations in real time. Sutherland implemented a one-pass relaxation method that propagated constraint satisfaction through the drawing. Constraint-based layout is now fundamental to UI frameworks (CSS’s box model, iOS’s Auto Layout, Android’s ConstraintLayout all implement variants of the concept), CAD software, and 3D modeling tools. All of them trace their conceptual lineage to Sketchpad.

The Object-Oriented Insight

The most intellectually consequential aspect of Sketchpad was not a user interface feature but a data structure: the way Sutherland organized the program’s internal representation of drawings.

Each element in a Sketchpad drawing was an object — a data record containing both the element’s properties (position, size, color) and the operations that could be applied to it (move, copy, constrain, instantiate). Master drawings were templates from which instances were created; each instance inherited the master’s structure but maintained its own position and transformation.

This organization — data bundled with the operations that act on it, templates from which instances are created, inheritance of structure from master to instance — is object-oriented programming. Sutherland arrived at it independently, from the specific needs of a graphical system, before any of the object-oriented programming languages had been formalized.

Alan Kay, who would go on to develop Smalltalk and the theoretical foundations of OOP, saw Sketchpad while a graduate student and described the experience as transformative. Kay’s Smalltalk, the language that named and systematized object-oriented programming, was a direct intellectual descendant of Sketchpad’s data model. The entire OOP tradition — C++, Java, Python, Ruby, Swift — traces its ancestry to a data structure Sutherland invented to represent drawings on a cathode-ray tube.

Head-Mounted Display and Virtual Reality

Sutherland completed his PhD in 1963 and moved to ARPA, where he succeeded J.C.R. Licklider as director of the Information Processing Techniques Office (IPTO) from 1964 to 1966. He was twenty-six when he took the role.

After ARPA he moved to Harvard, where in 1968 he built the first head-mounted display — a device that placed small screens in front of each eye, displaying a stereoscopic computer-generated image that changed as the user moved their head. The system was so heavy it had to be suspended from the ceiling by a mechanical arm; its creators called it the Sword of Damocles. The image it displayed was a simple wireframe room — a few lines representing the walls, floor, and ceiling of a virtual space that the user appeared to inhabit.

The Sword of Damocles was the first virtual reality system. It introduced the concepts that define VR sixty years later: stereoscopic display, head tracking, the experience of presence in a computer-generated environment. Sutherland’s 1968 paper describing it, “A Head-Mounted Three-Dimensional Display,” is the founding document of VR research.

“The Ultimate Display” (1965)

Three years before the Sword of Damocles, Sutherland published a paper called “The Ultimate Display” that described what a fully realized virtual reality system might eventually achieve: a room within which a computer could control the existence of matter, where a chair displayed in the room would be good enough to sit in and a bullet displayed in the room would be good enough to kill you. The paper is frequently cited as the most ambitious vision statement in computing history — a description of a goal so far beyond the technology of its era that it remained aspirational for half a century. It is also the foundational text of augmented and mixed reality research.

Evans & Sutherland: Building the GPU’s Ancestor

In 1968, Sutherland co-founded Evans & Sutherland with David Evans at the University of Utah. The company built specialized hardware for real-time 3D graphics — flight simulators, driving simulators, and eventually arcade game hardware.

The E&S hardware pioneered the pipeline architecture that underlies every modern GPU: geometry transformation, clipping, rasterization, and texturing implemented as dedicated hardware stages operating in sequence. A scene would enter the pipeline as a list of triangles and emerge as a framebuffer ready to display. The pipeline structure allowed hardware parallelism — each stage working on a different triangle simultaneously — that made real-time complex-scene rendering possible.

E&S flight simulators trained commercial and military pilots for decades. The company’s LDS-1 (Line Drawing System) and Picture System series established the geometry pipeline as the standard architecture for high-performance graphics. When NVIDIA, ATI, and others began building consumer 3D graphics hardware in the 1990s, they were implementing, in silicon and at vastly larger scale, an architecture whose fundamental structure Sutherland and Evans had worked out in the 1970s.

The University of Utah’s computer graphics program, which Sutherland helped shape, produced an extraordinary concentration of foundational contributors: Jim Clark (Silicon Graphics, Netscape), John Warnock (Adobe, PostScript), Alan Kay (Smalltalk, Xerox PARC), Ed Catmull (Pixar, computer animation), Henri Gouraud (Gouraud shading), Bui Tuong Phong (Phong shading and reflection model). The density of foundational work coming from one university department in one decade is unmatched in the history of computer science.

The Turing Award

Sutherland received the ACM Turing Award in 1988, citation:

“For his pioneering and visionary contributions to computer graphics, starting with Sketchpad, and continuing after.”

The citation understates the case. Sketchpad alone would justify the award. The head-mounted display alone would justify the award. The Evans & Sutherland hardware pipeline alone would justify the award. The training of a generation of computer graphics researchers at Utah would justify the award. Sutherland did all four before his fiftieth birthday.

Dead End: Sketchpad’s Premature Power

Sketchpad’s most important limitation was the machine it ran on. The TX-2 was a one-of-a-kind experimental computer at Lincoln Laboratory, not a commercial product. The program could not run anywhere else. Its capabilities — interactive graphics, constraint solving, hierarchical object structures — required hardware that commercial computers would not provide for nearly two decades.

The interactive computing revolution that Licklider had envisioned and that Sketchpad demonstrated had to wait for hardware to catch up. When Doug Engelbart demonstrated the NLS system in 1968 — with mouse, windows, and collaborative editing — he was working toward the same destination Sutherland had demonstrated in 1963, on hardware that was finally accessible enough to be the basis for commercial products.

The delay was not Sutherland’s failure; it was the inevitable consequence of being fifteen years ahead of the available hardware. But it meant that Sketchpad’s direct influence on commercial computing was mediated through researchers who had seen the demonstration or read the thesis, rather than through a product that users could experience directly. The ideas survived and propagated; the specific system was a historical artifact for most of the period between its demonstration and the arrival of personal computers.