Skip to content

Alex Krizhevsky

Abstract

Alex Krizhevsky trained the neural network that started the deep learning boom on two gaming graphics cards in his bedroom at his parents’ house in Toronto. AlexNet won the 2012 ImageNet competition by a margin large enough to end the argument about whether neural networks worked, and within a year most of the computer vision field had switched to his approach. He took the project on at the urging of a fellow graduate student. Five years later, after Google acquired the three-person company built around the result, he left, saying he had lost interest in the work. He is the least visible person in the history of modern AI, and he wrote its founding artifact.

Tiny Images

Krizhevsky studied at the University of Toronto, where Geoffrey Hinton supervised his graduate work.

His first widely used contribution was not a network but data. In 2009 he assembled CIFAR-10 and CIFAR-100, hand-labeled subsets of the 80 Million Tiny Images collection: 60,000 color images at 32 by 32 pixels, sorted into ten or a hundred classes. They were small enough to train on the hardware a graduate student could get and clean enough to compare methods on, and they became the standard benchmark for a decade of vision research. The accompanying technical report, Learning Multiple Layers of Features from Tiny Images, is one of the most cited documents in machine learning that was never published in a journal.

To train on them he wrote cuda-convnet, a convolutional network implementation in CUDA that ran on NVIDIA consumer graphics cards. This was unusual at the time. GPUs were for games and for a small amount of scientific computing; almost nobody in machine learning was writing their own CUDA kernels.

Being Talked Into It

GTX580 GF110 Chip
The GF110 graphics processor of the NVIDIA GeForce GTX 580, the consumer card of which Krizhevsky used two to train AlexNet. Image: Fritzchens Fritz, CC0, via Wikimedia Commons.

Ilya Sutskever, a fellow Hinton student, had concluded that neural networks would work if given enough data, and that Fei-Fei Li’s ImageNet was the dataset that would prove it. In 2011 he persuaded Krizhevsky, who already had a fast GPU network working on CIFAR-10, to scale it up to ImageNet’s 1.2 million images across a thousand categories.

The hardware was two NVIDIA GTX 580 cards, 3 gigabytes of memory each, in a machine in Krizhevsky’s bedroom at his parents’ house. That memory limit dictated the architecture: the network had to be split across the two cards, with the layers communicating only at certain points, because it did not fit on one. Krizhevsky extended cuda-convnet to handle the split and then spent roughly a year adjusting and retraining.

The finished network had eight learned layers, five convolutional and three fully connected, about 60 million parameters and 650,000 neurons. It used ReLU activations rather than the conventional sigmoid or tanh, which trained several times faster, and dropout in the fully connected layers to limit overfitting. A full training run took five to six days.

September 2012

The three entered the ImageNet Large Scale Visual Recognition Challenge as team SuperVision on September 30, 2012, and scored a top-5 error rate of 15.3 percent. The second-place entry, using the hand-engineered features that the field had spent fifteen years refining, scored 26.2 percent.

A margin of nearly eleven points in a competition where half a point was a result did not invite debate. Within a year essentially every serious computer vision group had switched to deep convolutional networks, and the industry began hiring the small number of people who knew how to train them. The consequences are traced in ImageNet and the Deep Learning Revolution and The GPU Revolution.

Krizhevsky presented the result at a computer vision workshop in Florence, Italy, in October 2012, alongside the European Conference on Computer Vision. The paper, ImageNet Classification with Deep Convolutional Neural Networks by Krizhevsky, Sutskever, and Hinton, appeared at NeurIPS in December 2012 and is now among the most cited papers in computer science.

Two details of that result mattered as much as the architecture. It ran on consumer gaming hardware, which meant any lab could reproduce it, and NVIDIA’s business changed permanently as a result. And it was trained by one graduate student, which meant the barrier to the next such result was low.

The Auction

Hinton, Sutskever, and Krizhevsky incorporated DNNresearch in late 2012, a company with three employees, no product, and no revenue, and let the large technology firms bid for it. The auction ran by e-mail in December 2012 from room 731 of Harrah’s casino hotel at Lake Tahoe, where the NeurIPS conference was being held that year. Four bidders took part: Google, Microsoft, Baidu, and DeepMind. Hinton stopped the bidding at $44 million and chose Google; the deal closed on 12 March 2013, and the three joined the company.

Krizhevsky worked at Google for about four and a half years, on image recognition for Google Photos and then on the company’s self-driving car project. In September 2017 he left; he had lost interest in the work, he told Quartz in 2018. He went to Dessa, a small Toronto machine learning company later acquired by Square, and has kept largely out of public view since, while Hinton collected a Nobel Prize and Sutskever co-founded OpenAI.

The 2012 AlexNet source code was released by Google and the Computer History Museum on March 20, 2025, after five years of negotiation in which Hinton helped connect the museum with the right people at Google. Hinton summed up the division of labor in the museum’s announcement: “Ilya thought we should do it, Alex made it work, and I got the Nobel Prize.”

What It Was and Was Not

AlexNet introduced no fundamentally new idea. Convolutional networks were Yann LeCun’s, from the 1980s; backpropagation was older; ReLU and dropout were recent but not Krizhevsky’s alone. What he supplied was the demonstration, at a scale nobody had attempted, that the old approach worked when given enough data and enough arithmetic.

Read this way, the long period in which neural networks were out of favor had not been caused mainly by a missing idea. Machines had been too slow and datasets too small to show that the existing ideas worked, and the demonstration came when a graduate student split a network across two gaming cards in his bedroom and trained it on 1.2 million labeled images.

📚 Sources