A Brief History of AI – LLMs
When I wrote about “The Next Tech Revolution Isn’t AI” a year ago, and my view on releasing us from smartphones, the point that stuck with readers was that the phone in your pocket was built on the shoulder of giants but arrived at a specific point in time. Lithium-ion batteries, capacitive touchscreens, cellular radio, GPS: none of it was invented by the companies whose logos are on the back of the sleek, sophisticated phones.
The same is true of the AI models everyone now uses.
My reasons for writing this post are partly selfish. I’ve been delivering AI projects since 2019, and I’ve kept reaching for these dates and names in conversations, articles and client work, and I kept having to look them up, so I wanted one page I could refer back to.
But I also wanted to put the timeline together for a different reason. The reader I have in mind is the person who’s AI journey began on, or after 30 November 2022, the day ChatGPT officially launched. If that’s you, you arrived at the end of a very long film, and most of what you’ve seen since has been about the last, action-packed ten minutes.
This post is the backfill: the giants, the decisions that turned out to matter, the stalls, and the innovations written off as ahead of their time, some of them twice. It’s also a way to see how long the people now running the frontier labs have been at this. The names at the top of the field were doing this work in 2012, in 1997, and in one case in 1986.
How to read it. Each entry has a plain description of what the thing was, the innovation, why it mattered to what came later, and a link to the the closest primary source I can find. The entries are grouped into five eras, because the eras are the story: eighty years of ideas that were mostly ignored, then a decade in which the compute, the data and the algorithms all arrived at once.
Where something stalled, I’ve said what stalled it. Where a date is contested, I’ve said that too. I’ve kept to the innovations that a modern large language model (LLM) directly depends on, plus the hardware and data work that made them practical; speech, robotics and most of computer vision are their own stories with their own giants.
Almost nothing on this list was built by a frontier lab. Most of it was built by people who were told they were wasting their time.
Foundations: 1943 to 1989
The first era is a set of ideas that arrived long before the hardware to run them. Every one of them was sound, and every one of them stalled, usually for want of compute, sometimes for want of belief. The pattern that repeats through the whole timeline shows up here first: a good idea, a demonstration on a problem, and then a wait.
1943: the artificial neuron
Warren McCulloch and Walter Pitts described a neuron as a simple logic unit: add up the inputs, fire if they cross a threshold. It’s a mathematical cartoon of a brain cell, but it was the first time anyone showed that networks of such units could compute anything a logic circuit could.
Why it mattered: every neural network since, including the ones behind today’s models, is a descendant of this abstraction.
1948: information theory and the first statistical language model
Claude Shannon’s paper on communication gave us the bit, entropy and the mathematics of how much information a message carries. Tucked inside it is a demonstration of generating English by predicting each letter or word from the ones before it: the n-gram model.
Why it mattered: predicting the next token from context is exactly what a language model does. Shannon set the problem and the yardstick 75 years before GPT-3 solved it well.
1949: Hebbian learning
Donald Hebb proposed that when one neuron repeatedly helps fire another, the connection between them strengthens. It’s usually shortened to “cells that fire together wire together”.
Why it mattered: it was the first credible account of how a network could learn from experience by adjusting connection strengths, which is the basic move in all machine learning.
1950: Turing asks whether machines can think
Alan Turing’s paper sidestepped the philosophy and proposed a test: can a machine hold a text conversation well enough that a person can’t tell it from a human? He also predicted machines would need to learn rather than be programmed, and suggested starting with a “child machine” and educating it.
Why it mattered: he framed the goal as conversation, and he framed the method as learning. Both turned out to be right.
1956: the Dartmouth workshop names the field
John McCarthy, Marvin Minsky, Nathaniel Rochester and Claude Shannon submitted a proposal on 31 August 1955 for a summer 1956 workshop at Dartmouth College, built around the conjecture that “every aspect of learning or any other feature of intelligence” could in principle be described precisely enough for a machine to simulate it. McCarthy used the term “artificial intelligence” in the proposal.
Why it mattered: it gave the field its name and its founding optimism, and it set the symbolic, rule-based approach that would dominate for thirty years while the neural-network tradition went in and out of favour beside it.
1958: the Perceptron
Frank Rosenblatt built the first neural network that learned from examples. Show it labelled patterns, let it adjust its weights when it gets one wrong, and it improves. The New York Times reported the Navy expected it to walk, talk and reproduce itself.
Why it mattered: it worked, on simple problems, and the hype around it set up the backlash that followed.
1965: Moore's Law
Gordon Moore observed that the number of components on a chip was doubling roughly every year (later revised to two) and predicted it would continue. It did, for half a century.
Why it mattered: nearly every “breakthrough” in this timeline is an old idea meeting new hardware. Moore’s curve is what delivered the hardware.
1969: Perceptrons, and the first winter
Marvin Minsky and Seymour Papert proved that a single-layer perceptron can’t represent some simple functions, including exclusive-or. Multi-layer networks could, but nobody had a practical way to train them. Funding for neural networks dried up for most of the next fifteen years, and the book took most of the blame, not all of it fairly.
Why it mattered: the mathematics was right. The episode is the first example on this list of a correct result about today’s implementation being heard as a verdict on the whole approach.
1970 to 1974: reverse-mode differentiation and backpropagation
Seppo Linnainmaa published an efficient form of reverse-mode automatic differentiation in 1970, without talking about neural networks. Paul Werbos’s 1974 Harvard PhD thesis then described how the same basic backwards flow of derivatives could be used to optimise multi-stage adaptive systems, including neural networks. Neither was read much by the people who needed it.
Why it mattered: reverse-mode differentiation is the mathematical engine under modern backpropagation. It’s how a deep network works out how much each of millions or billions of parameters contributed to an error and adjust them in the useful direction.
1980: the Neocognitron
Kunihiko Fukushima built a layered network inspired by the visual cortex, with small detectors scanned across an image and the results pooled together. It could recognise handwritten characters regardless of where they sat on the page.
Why it mattered: it’s the ancestor of the convolutional network, the architecture that won ImageNet in 2012 and made the world take deep learning seriously.
1986: backpropagation, rediscovered and believed
David Rumelhart, Geoffrey Hinton and Ronald Williams published a short paper in Nature showing that backpropagation lets a multi-layer network learn useful internal representations. The idea wasn’t new. The demonstration, and the authors’ persistence in promoting it, were.
Why it mattered: this is the paper that ended the first winter. It’s also the start of Hinton’s forty-year argument with the rest of the field.
1989: LeCun reads postcodes
At Bell Labs, Yann LeCun applied backpropagation to a convolutional network and trained it on handwritten digits from US Postal Service envelopes. The follow-on system, LeNet-5, was published in 1998 and ended up reading a meaningful share of the cheques processed in the United States.
Why it mattered: the first neural network doing real work at scale. It proved the recipe of convolution plus backprop plus data, twenty years before the hardware made it general.
The long winter: 1989 to 2009
Ask most people when deep learning started and they’ll say 2012. Ask the people who were there and they’ll point to this stretch, when the money went to other methods and a small group kept working anyway. Four things happened in these twenty years that the 2012 breakthrough depended on:
- reinforcement learning proved itself on a real game
- the sequence-modelling problem got a workable answer
- GPUs turned from graphics cards into general-purpose computers
- the internet quietly became the largest text corpus ever assembled
1989 to 1995: Q-learning and TD-Gammon
Chris Watkins’s 1989 thesis introduced Q-learning, a way for an agent to learn which actions are valuable from delayed rewards; Watkins and Peter Dayan published the convergence result in 1992. At IBM, Gerald Tesauro used temporal-difference learning and self-play to build TD-Gammon. The first version reached near-expert play; later versions held their own against the world’s best.
Why it mattered: TD-Gammon was the first proof that a neural network trained by reinforcement and self-play could reach top human level at anything. AlphaGo, AlphaZero and the reinforcement learning that tunes today’s chat models follow the same recipe.
1990: statistical machine translation
A team at IBM including Peter Brown showed you could translate French into English with almost no grammar rules, by learning probabilities from the Canadian parliament’s bilingual proceedings. Frederick Jelinek, who ran the group, is remembered for the line that every time he fired a linguist, the system got better.
Why it mattered: it established that for language, data beats hand-written rules. That instinct runs straight through to today’s models.
1991 to 1997: the vanishing gradient and the LSTM
Sepp Hochreiter’s 1991 diploma thesis identified why deep and recurrent networks were so hard to train: the error signal fades to nothing as it passes back through layers. In 1997 he and Jürgen Schmidhuber published the Long Short-Term Memory network, which adds gated memory cells that let a signal persist across long sequences.
Why it mattered: LSTMs were the dominant way to model text, speech and time series for two decades. Google’s translation and speech systems ran on them until the Transformer arrived.
Hochreiter and Schmidhuber, “Long short-term memory”, Neural Computation, 1997
1999: the GPU gets a name
NVIDIA announced the GeForce 256 in August 1999 and shipped it in October, marketing it as the world’s first “GPU” or graphics processing unit. Graphics processors existed before the name did, but this generation pulled more of the 3D pipeline onto specialised hardware designed to carry out many similar calculations in parallel.
Why it mattered: “the same arithmetic on huge numbers of things at once” is also a description of a neural network. The games industry paid for the chips that AI would later run on.
2001: Wikipedia
Wikipedia launched in January 2001. Within a decade it was the largest, cleanest, most broadly licensed body of general knowledge text in existence, written by volunteers.
Why it mattered: it’s in the training data of essentially every large language model, and it’s a reminder that a lot of what a model knows is the digitised labour of people who were never paid for it and never expected to train a machine.
2003: neural language models and word embeddings
Yoshua Bengio’s group trained a neural network to predict the next word, and in doing so it learned to represent each word as a list of numbers where similar words sit close together. That representation, the embedding, was learned rather than designed.
Why it mattered: it showed a neural network could beat the n-gram models that had held the field since Shannon, and it introduced the idea that meaning could be a position in a vector space.
Bengio et al., “A neural probabilistic language model”, Journal of Machine Learning Research, 2003
2005: Mechanical Turk
Amazon launched a marketplace for paying people small sums to do small tasks that computers couldn’t. Labelling images was the obvious one.
Why it mattered: ImageNet’s fourteen million labels were produced by nearly 50,000 workers on this platform. No labels, no ImageNet; no ImageNet, no 2012.
2006: deep belief networks and the return of "deep learning"
Hinton, with Simon Osindero and Yee-Whye Teh, showed a way to train networks many layers deep by pretraining each layer on its own before fine-tuning the whole stack. The results on handwritten digits beat everything else.
Why it mattered: it broke the belief that deep networks were untrainable in practice, and it revived the term “deep learning” as the banner for the movement.
Hinton, Osindero and Teh, “A fast learning algorithm for deep belief nets”, Neural Computation, 2006
2006 to 2007: CUDA
NVIDIA announced CUDA alongside its G80 chip in November 2006 and shipped it to developers in 2007. It let programmers write ordinary C code to run on a graphics card, with no need to disguise the work as a rendering job.
Why it mattered: this is the moment GPUs became general-purpose computers. Every deep learning framework since has been built on it or on its imitators.
2007 to 2008: Common Crawl
Gil Elbaz, who had sold his previous company to Google, founded a non-profit to crawl the open web and give the archive away free. Regular crawls began in 2008.
Why it mattered: Common Crawl is the raw material for most of the text that language models are trained on. A single philanthropic decision made web-scale training data available to anyone.
2009: ImageNet
Fei-Fei Li, then at Princeton, with Jia Deng and colleagues, published a dataset of millions of photographs labelled against a hierarchy of tens of thousands of nouns. The prevailing view was that better algorithms mattered and bigger datasets didn’t. She built it anyway, and from 2010 ran an annual competition on it.
Why it mattered: it gave the field a benchmark hard enough that only deep learning could win it, and it made the case that data is as much a research contribution as an algorithm.
Deng et al., “ImageNet: a large-scale hierarchical image database”, CVPR, 2009
2009: deep learning on GPUs
Rajat Raina, Anand Madhavan and Andrew Ng at Stanford showed that training deep networks on a GPU was up to 70 times faster than on a CPU, turning weeks into a day.
Why it mattered: it’s the paper that connected the last two decades of graphics hardware to the next decade of AI.
The explosion: 2012 to 2016
Here the three threads meet. Convolution plus backprop from the eighties, gaming GPUs from the nineties, and a dataset from 2009, all in one system, trained by a graduate student in his bedroom. What follows is the fastest five years in the history of the field, and it’s worth noticing how much of it was done by a handful of groups in Toronto, Montreal, Stanford and London before the big money arrived.
2012: AlexNet
Alex Krizhevsky, with Ilya Sutskever and Geoffrey Hinton, entered a deep convolutional network in the ImageNet competition and trained it on two NVIDIA GTX 580 GPUs. Their ensemble achieved a top-five error rate of 15.3%, against 26.2% for the second-best entry: a gap large enough that nobody could dismiss it as another marginal benchmark improvement.
Why it mattered: this is the result that convinced industry. Google acquired the team’s DNNresearch company in 2013; Facebook recruited LeCun to build FAIR; Baidu brought Ng in as chief scientist in 2014. From here the work happens at corporate scale.
2012 to 2014: dropout and Adam
Two pieces of training plumbing became workhorses. Hinton and colleagues introduced dropout in 2012, randomly removing units during training so the network could not depend too heavily on particular paths; the fuller JMLR treatment followed in 2014. In late 2014, Diederik Kingma and Jimmy Ba introduced Adam, an adaptive optimiser that made effective training much less sensitive to the choice of a single global learning rate.
Why it mattered: unglamorous plumbing, and the reason training a large model went from a craft practised in a few laboratories to something you could repeat.
2013: word2vec
Tomas Mikolov’s team at Google published a fast, simple way to learn word embeddings from raw text. The famous demonstration was arithmetic on meaning: king minus man plus woman lands near queen.
Why it mattered: Bengio’s 2003 idea became practical at web scale, and every language model since has started by turning words into vectors this way.
2013: deep reinforcement learning plays Atari
DeepMind, then a small London startup, trained a single network to play seven Atari games from raw pixels using nothing but the score as feedback. On several it beat expert humans. The Nature version, in 2015, covered 49 games.
Why it mattered: it joined deep learning to the reinforcement learning tradition from TD-Gammon, and the paper is the reason Google paid around half a billion dollars for the company in 2014.
2014: sequence to sequence, and attention
Sutskever, Oriol Vinyals and Quoc Le showed that two LSTMs, one to read a sentence and one to write the translation, could match the best machine translation systems with no linguistic rules. Within weeks Dzmitry Bahdanau, Kyunghyun Cho and Bengio fixed its main weakness (squeezing a whole sentence into one vector) by letting the writer look back at any part of the input as it went. They called the mechanism attention.
Why it mattered: attention is the idea the Transformer is built on. This is where it was born, as a patch for translation.
2015: ResNet and open frameworks
Kaiming He and colleagues at Microsoft Research Asia found that adding “skip connections”, which let a signal bypass a layer, made networks of 100-plus layers trainable. Their 152-layer ResNet won ImageNet. The same year Google open-sourced TensorFlow; Facebook’s PyTorch followed in early 2017.
Why it mattered: residual connections are inside every Transformer block. And the frameworks meant a graduate student anywhere could build what had needed a lab.
2015 to 2016: subword tokenisation
Rico Sennrich, Barry Haddow and Alexandra Birch adapted byte-pair encoding to neural machine translation, breaking rare or unseen words into reusable subword pieces instead of requiring an impossibly large vocabulary containing every whole word.
Why it mattered: language models don’t read whole words; they read tokens, which are words, fragments of words and punctuation, and this is where that convention comes from. Without it, a model with an open vocabulary wasn’t practical.
2016: AlphaGo, and the TPU
In March, DeepMind’s AlphaGo beat Lee Sedol, one of the strongest Go players alive, four games to one. Go had been expected to resist computers for another decade. In May, Google announced it had been running a custom chip for neural networks, the Tensor Processing Unit, in its data centres for a year.
Why it mattered: AlphaGo was the public moment, watched by 200 million people. The TPU was the quieter one: the first sign that the frontier would be decided by who could build their own silicon.
The language model era: 2017 to 2020
You can date the modern era to one paper. But the paper only works because of two things that happened around it: a recipe for teaching models what humans prefer, and the discovery that if you just make the thing bigger, it keeps getting better in a way you can plot on a graph.
2017: the Transformer
Eight researchers at Google, listed on the paper as equal contributors (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin), threw away recurrence and convolution and built a model out of nothing but attention. It read every position in a sequence in relation to every other at once, which meant it could be trained in parallel on the GPUs and TPUs that had been waiting for exactly this.
Why it mattered: it’s the T in GPT. Every frontier model is a Transformer, and its central ideas (attention, feed-forward layers, residual pathways) are still recognisable under everything that has been bolted on since: routing, mixtures of experts, multimodality.
2017: learning from human preferences, and PPO
Paul Christiano and colleagues at OpenAI and DeepMind showed you could train a reinforcement learning agent from human comparisons (“this one is better than that one”) rather than a hand-written reward. A month later John Schulman’s team published Proximal Policy Optimization, a stable, simple way to do the reinforcement step.
Why it mattered: put the two together and you have RLHF, the technique that turned GPT-3 into ChatGPT.
2018: pretrain, then fine-tune
Four papers in one year established the recipe. ELMo and ULMFiT showed that a language model trained on unlabelled text could be adapted cheaply to specific tasks. OpenAI’s GPT applied the same idea to a Transformer. Google’s BERT did it bidirectionally and broke every benchmark in natural language processing.
Why it mattered: this is the point at which “train once on the internet, then specialise” became the default, and the point at which model size started to look like the main lever.
- Peters et al., “Deep contextualized word representations” (ELMo), 2018
- Howard and Ruder, “Universal language model fine-tuning for text classification” (ULMFiT), 2018
- Radford et al., “Improving language understanding by generative pre-training” (GPT), 2018
- Devlin et al., “BERT: pre-training of deep bidirectional transformers for language understanding”, 2018
2019: GPT-2 and the bitter lesson
OpenAI’s GPT-2, at 1.5 billion parameters, wrote coherent multi-paragraph text with no task-specific training at all, and the company initially withheld the full model on safety grounds. A month later Richard Sutton, one of the founders of reinforcement learning, published a short essay arguing that seventy years of AI history teach one thing: methods that use more compute always beat methods that use more human cleverness.
Why it mattered: GPT-2 was the first glimpse of a general text model. Sutton’s essay is the philosophy the frontier labs have run on since.
2020: scaling laws and GPT-3
In January, Jared Kaplan’s team at OpenAI showed that a language model’s performance improves as a smooth, predictable function of model size, data and compute: draw the line and you can forecast the next model. In May, GPT-3 arrived at 175 billion parameters and did something nobody had designed it to do. Give it a task description and a handful of examples in the prompt, or even a single one, and it would do the task. They called it few-shot and one-shot learning.
Why it mattered: scaling laws gave the labs a business case for spending hundreds of millions on a single training run. GPT-3 showed that prompting was a new way to program a computer.
Making it usable: 2022 to 2023
GPT-3 was a remarkable engine with no steering wheel. The last stretch of this timeline is about steering: eliciting useful reasoning, following human instructions, using training compute more efficiently, and putting the result behind an interface normal people could understand. I stop the main chronology in 2023 deliberately. After that, the field becomes current history: multimodal models, agents, reasoning systems, enormous infrastructure builds and a rapidly changing market whose significance is still being argued over.
2022: chain of thought, then "let's think step by step"
Jason Wei and colleagues at Google showed that giving sufficiently large language models worked examples containing intermediate reasoning steps could dramatically improve performance on some arithmetic, symbolic and commonsense problems. A few months later, Takeshi Kojima and colleagues found that even a single zero-shot instruction, “Let’s think step by step”, could produce large gains on several reasoning benchmarks.
Why it mattered: the reasoning had been in there; it just needed asking for. This is the seed of today’s “reasoning” models, which spend compute thinking before they answer.
2022: InstructGPT
OpenAI applied the 2017 preference-learning recipe to GPT-3. Human contractors wrote ideal answers and ranked model outputs; the rankings trained a reward model; PPO tuned the language model against it. A 1.3 billion parameter model trained this way was preferred by people over the raw 175 billion parameter GPT-3.
Why it mattered: this is the step that made the models follow instructions instead of just continuing text. ChatGPT is InstructGPT with a chat interface.
2022: Chinchilla
DeepMind’s Jordan Hoffmann and colleagues revisited scaling and found that many large language models were being trained with too few tokens for their parameter count. Chinchilla used about the same training compute as the much larger Gopher but spread that compute across a smaller model and far more data, and it outperformed Gopher on a range of tasks.
Why it mattered: it re-priced the whole endeavour. Data, not parameters, became the constraint, which is why the scramble for training text has been so intense since.
30 November 2022: ChatGPT
OpenAI put an RLHF-trained GPT-3.5-series model behind a conversational interface and released it as a free research preview. The model could remember the thread of a dialogue, answer follow-up questions, respond to corrections and refuse some requests. The public did not have to learn an API, prompt syntax or machine-learning workflow. They could type.
Why it mattered: there was no new foundational architecture underneath it, but the combination of conversational post-training, instruction following and an interface anyone could use changed the world’s perception of what the existing technology was capable of. The interface was not cosmetic. It was the moment the capability became legible.
2022 to 2023: Constitutional AI, GPT-4, and the open-weight ecosystem
In December 2022 Anthropic published Constitutional AI, using a written set of principles and AI-generated critique and feedback to reduce reliance on human preference labels for some parts of alignment. In February 2023 Meta released LLaMA to approved researchers; its weights leaked within a week and accelerated an already growing open-model community. In March, OpenAI released GPT-4, showing a substantial jump in capability and adding image input to the GPT line.
Why it mattered: between them they set the shape of the current market: closed frontier models, open-weight alternatives close behind, and post-training as a competitive layer in its own right.
The other branches
A reader who arrived in 2022 could be forgiven for thinking AI means language models. The timeline above follows one line because that’s the line that leads to the chat box, but four other branches share most of the same roots, and some of them are further along. Each gets a paragraph and a few pointers here; in reality each deserves its own page.
Computer vision after ImageNet
Vision is where deep learning proved itself, and it kept moving after 2012. Ross Girshick’s R-CNN work in 2013 to 2014 brought deep convolutional features decisively into object detection: not just “there’s a dog” but “the dog is here”. YOLO in 2016 made detection fast enough for real-time use. U-Net in 2015 became a workhorse for segmenting structures in biomedical images. In 2020 Google researchers showed that a Transformer could replace convolution for image recognition too, and in 2021 OpenAI’s CLIP learned a shared representation of images and language, enabling systems to relate pictures to natural-language descriptions. On the generative side, diffusion models described by Jonathan Ho and colleagues in 2020 overtook GANs within a year, and latent diffusion, the method behind Stable Diffusion, put them on consumer hardware in 2022.
- Girshick et al., “Rich feature hierarchies for accurate object detection and semantic segmentation” (R-CNN), 2013/14
- Redmon et al., “You only look once: unified, real-time object detection” (YOLO), 2015/16
- Ronneberger, Fischer and Brox, “U-Net”, 2015
- Dosovitskiy et al., “An image is worth 16×16 words” (Vision Transformer), 2020
- Radford et al., “Learning transferable visual models from natural language supervision” (CLIP), 2021
- Ho, Jain and Abbeel, “Denoising diffusion probabilistic models”, 2020
- Rombach et al., “High-resolution image synthesis with latent diffusion models”, 2021/22
Speech
Statistical speech recognition became commercially important long before today’s chatbots, with Hidden Markov Models dominating the field for decades. In 2012 a joint paper from Hinton’s group with Google, Microsoft and IBM showed deep networks making substantial cuts in error rates. Baidu’s Deep Speech in 2014 trained end to end from audio to text with no hand-built phonetic pipeline. OpenAI’s Whisper in 2022 applied the Transformer recipe to multilingual speech and made near-human transcription a free download. Every voice assistant you’ve used sits on this branch.
Symbolic AI
For the first thirty years after Dartmouth, AI mostly meant representing knowledge explicitly and searching through rules or possibilities. Newell, Simon and Shaw’s Logic Theorist proved mathematical theorems in the 1950s. Joseph Weizenbaum’s ELIZA held a conversation in 1966 by pattern-matching and reflecting the user’s own words back, and Weizenbaum was disturbed by how readily people confided in it. Terry Winograd’s SHRDLU in 1970 could respond to English commands about a tightly constrained world of blocks. IBM’s Deep Blue beat reigning world chess champion Garry Kasparov in 1997 by combining specialised hardware, an evaluation function, databases and extremely fast search, around 200 million positions a second, rather than learning during the match. Doug Lenat’s Cyc project spent from 1984 until his death in 2023 hand-coding millions of common-sense facts, the opposite bet to today’s models. Symbolic AI lost its monopoly rather than its life: search, planning, tools and structured knowledge now sit alongside learned models again.
Reinforcement learning beyond games
The TD-Gammon line did not stop at Go. AlphaZero, reported in 2017 and published in Science in 2018, learned chess, shogi and Go from the rules alone, with no human games, and beat the best existing programs at all three. The same organisation’s AlphaFold2 system all but solved the fifty-year-old problem of predicting a protein’s shape from its sequence; Demis Hassabis and John Jumper later shared half of the 2024 Nobel Prize in Chemistry for protein structure prediction, with the other half awarded to David Baker for computational protein design. In robotics, OpenAI trained a dexterous hand to solve a Rubik’s cube in 2019 using large amounts of simulation and domain randomisation. And reinforcement learning has come back to language models, in post-training on maths and code where an answer can be checked, which is where the current “reasoning” models come from.
- Silver et al., “A general reinforcement learning algorithm that masters chess, shogi and Go through self-play”, Science, 2018
- Jumper et al., “Highly accurate protein structure prediction with AlphaFold”, Nature, 2021
- Nobel Prize in Chemistry 2024
- OpenAI, “Solving Rubik’s cube with a robot hand”, 2019
The cloud
There’s a whole branch missing from this page, and it’s the one I spent most of my career in. Nothing above runs without the cloud.
GPT-3 was trained on a supercomputer Microsoft built inside Azure for the purpose; every frontier model since has been trained in someone’s data centre and served from a fleet of them, and the reason a chat box answers you in seconds from anywhere on earth is thirty years of work on global, resilient infrastructure that has nothing to do with neural networks.
Virtualisation (VMware in 1999, Xen in 2003), the arrival of rentable compute with Amazon’s S3 and EC2 in 2006, containers with Docker in 2013 and Kubernetes in 2014, and underneath all of it the networking, the fibre, the power and the cooling that let a model trained in one building be used from every other.
That story has its own giants, its own stalls and its own written-off ideas, and it deserves its own page rather than a paragraph.
A Brief History of the Cloud is on the list. For now, take it as read that when this timeline says “at scale”, it’s leaning on all of it.
Fallen by the wayside
Every timeline flatters the survivors. Here are the ideas and people the main list steps over, in three kinds: the approaches that dominated and then vanished from the frontier, the people who were right too early, and the decisions that stopped the field in its tracks. The uncomfortable point is that, at the time, nobody could reliably tell the first kind from the second.
Dominant, then gone
Expert systems were the AI boom before this one. DENDRAL and MYCIN in the 1960s and 70s encoded specialist knowledge as rules; XCON at Digital Equipment configured computer orders from 1980 and was credited with saving the company tens of millions of dollars a year. A whole industry of consultancies and specialist Lisp machines grew around expert systems, and it collapsed in 1987 when cheaper workstations arrived and the systems proved too brittle and too expensive to maintain. Japan’s Fifth Generation Computer Systems project, launched by MITI in 1982, ran for ten years pursuing logic-programming machines and produced little that outlived it. Support vector machines, formalised by Corinna Cortes and Vladimir Vapnik in 1995, beat neural networks on most benchmarks for fifteen years and had the better theory; they’re one reason Hinton’s students struggled to get papers accepted in the 2000s. Hidden Markov Models owned speech recognition for decades. GANs, Ian Goodfellow’s 2014 idea of training a generator against a discriminator, owned image generation until diffusion models beat them in 2021. Seven years from invention to obsolescence tells you something about the speed of the current period.
Right too early
Seppo Linnainmaa published the mathematics behind reverse-mode automatic differentiation in 1970, in work that was not about neural networks. Frank Rosenblatt died in a boating accident in 1971, two years after Minsky and Papert’s book and fifteen before the revival that vindicated him. Fukushima’s Neocognitron had the convolutional architecture in 1980 but no backprop to train it. Werbos had backprop in 1974 but no audience. Hochreiter had the vanishing-gradient problem diagnosed in a 1991 German-language thesis. Even the giants have wayside ideas: Hinton’s capsule networks in 2017 were his own proposed successor to the convolutional networks he’d helped make famous, and the field went another way. IBM’s Watson beat two of the strongest Jeopardy! champions in 2011 and was then sold to the world as the future of medicine; IBM sold off Watson Health in 2022 and it became Merative. Being first is not the same as being the one who gets the credit, and being famous is not the same as being right.
The decisions that stopped the clock
Three stalls have names on them, though none has a single cause. Minsky and Papert’s Perceptrons in 1969, covered above, took American funding away from neural networks for a decade and a half, or was blamed for it, which amounted to the same thing. In Britain, James Lighthill was asked by the Science Research Council to review AI research in 1972; his report, published in 1973, concluded that AI had delivered on none of its promises, and British funding for the field all but vanished for a decade; historians still argue about how much of that was the report and how much the mood it caught. The BBC televised a Royal Institution debate in which Lighthill defended his conclusions against Donald Michie, John McCarthy and Richard Gregory. Then the expert-systems boom broke in 1987, DARPA’s Strategic Computing programme was cut back, and the second winter set in, the one Hinton, LeCun and Bengio spent their early careers in. Each was a reasonable call on the evidence and the economics visible at the time, which is what makes them worth studying rather than mocking.
The people who were already there
If you only know these names from the last few years of headlines, it’s worth seeing where each first appears on the list above. The frontier didn’t hire its leaders in 2022. They’d been building it, mostly without much attention, for decades.
Geoffrey Hinton
First appears here in 1986 with backpropagation, again in 2006 with deep belief nets, and in 2012 as AlexNet’s senior author. Google acquired the DNNresearch company he formed with Krizhevsky and Sutskever in 2013. He left Google in 2023 and shared the 2024 Nobel Prize in Physics with John Hopfield “for foundational discoveries and inventions that enable machine learning with artificial neural networks”. Nearly forty years on the one bet: that what a system knows can be learned rather than written down by hand.
Yann LeCun
Reading handwritten ZIP codes at Bell Labs in 1989. He founded Facebook AI Research in 2013 and later became Meta’s Chief AI Scientist. He left Meta at the end of 2025 to build Advanced Machine Intelligence, or AMI Labs, where he is Executive Chairman. In March 2026 AMI announced a $1.03 billion funding round for an approach centred on world models, reasoning and planning rather than simply scaling language models.
Yoshua Bengio
The 2003 neural language model and the 2014 attention paper. He, Hinton and LeCun shared the 2018 ACM A.M. Turing Award for conceptual and engineering breakthroughs that made deep neural networks a critical component of computing.
Sepp Hochreiter and Jürgen Schmidhuber
The 1991 thesis and the 1997 LSTM. Schmidhuber has spent a good part of the last decade arguing, with some justice, that his lab’s work from the nineties is under-credited in the story the field tells about itself. Whatever you make of the credit dispute, the pre-Transformer era ran on LSTM.
Fei-Fei Li
ImageNet, 2009, built against the advice of her senior colleagues. She later served as Chief Scientist of AI/ML at Google Cloud, co-founded Stanford’s Institute for Human-Centered AI, and in 2024 co-founded World Labs with Justin Johnson, Ben Mildenhall and Christoph Lassner to work on spatial intelligence and world models.
Andrew Ng
The 2009 GPU paper. He was the founding lead of Google Brain in 2011, became Baidu’s Chief Scientist in 2014, and co-founded Coursera. His influence on the field comes from both research and an unusual ability to turn machine learning into something enormous numbers of people could actually learn.
Demis Hassabis
Co-founded DeepMind in 2010, five years before OpenAI existed. The Atari paper in 2013, AlphaGo in 2016, and then AlphaFold. In 2024 he and John Jumper shared half of the Nobel Prize in Chemistry for protein structure prediction, with David Baker receiving the other half for computational protein design.
Ilya Sutskever
AlexNet in 2012, sequence to sequence in 2014, then co-founder and Chief Scientist of OpenAI from 2015. He left OpenAI in May 2024 and co-founded Safe Superintelligence the following month. He has been its chief executive since July 2025.
John Schulman
Proximal Policy Optimization in 2017, an OpenAI co-founder, and, in his own description, the person who led the creation of ChatGPT and later co-led OpenAI’s post-training team. He left OpenAI for Anthropic in 2024, then joined Mira Murati’s Thinking Machines Lab as co-founder and Chief Scientist.
Dario Amodei
A co-author on the 2020 scaling-laws paper and on GPT-3, after earlier work at Baidu and Google Brain. He left OpenAI at the end of 2020 and co-founded Anthropic with Daniela Amodei and other former OpenAI researchers; he remains Anthropic’s Chief Executive.
The eight Transformer authors
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, listed on the paper as equal contributors. By 2023 all eight had left Google, many to found companies of their own, including Cohere, Character.AI, Inceptive, Essential AI and Sakana AI. Google brought Noam Shazeer back in 2024 as part of a Character.AI licensing-and-hiring deal reported at roughly $2.7 billion; in June 2026 he left Google again to join OpenAI. The single most valuable paper of the era, and the company that funded it kept none of its authors.
What the timeline says
Look down the list before 2012 and today’s frontier labs don’t exist. Even after they arrive, most of the building blocks come from universities, industrial research labs, open-source communities, non-profits and work done for entirely different markets. That’s not a complaint about the labs; scale, infrastructure and the engineering to make them work are real contributions. It matters because the public story of AI is easily compressed into a few companies and a few years, while the record is a story about a few hundred people and eighty years. If you arrived after ChatGPT, that’s the thing to take away: the explosion you witnessed was the last visible flight of a staircase, and the staircase is the interesting part.
There’s a second thing. Nearly every entry from before 2012 was, at the time, a bad career move. Werbos wrote backpropagation into a thesis nobody read. Hinton spent the winter funded by the Canadian Institute for Advanced Research because nobody else would pay. Li has described being told that building a dataset was not serious research. Hochreiter’s diagnosis of the vanishing-gradient problem sat in a German-language diploma thesis before LSTM gave it a practical answer. Plenty of their unfashionable ideas lost. What they shared was the ability to hold a clear idea of the problem they wanted to solve for longer than the surrounding fashion lasted. Henry Maudslay’s maxim, recorded by his apprentice James Nasmyth, was: “First, get a clear notion of what you desire to accomplish.” That feels as relevant to this history as any equation in it.
Which brings me to the thing I can’t yet resolve. Jack Kilby demonstrated an integrated circuit at Texas Instruments in 1958; Robert Noyce independently developed the practical monolithic silicon approach at Fairchild in 1959. It still took years of fabrication, standards, tooling, design practice and falling costs before integrated circuits became ordinary components that engineers could build whole industries around. If the analogy holds, AI may still be somewhere around its 1958 comparison to electronics. The work that gets it to its 1978, the reliability, governance, evaluation, interfaces, operating practices and tooling that let an ordinary team build with it safely, is only partly visible in this timeline because much of it is still being invented.
I don’t know who the giants of that stretch will be. I’d like to think some of them are reading.
Frequently asked questions
Who invented backpropagation?
It depends what you count. Seppo Linnainmaa published the underlying reverse-mode automatic-differentiation method in 1970. Paul Werbos was an early and explicit advocate of applying backwards differentiation to train neural and adaptive systems in his 1974 thesis. Rumelhart, Hinton and Williams gave backpropagation the demonstration and audience that made it central to neural-network research in 1986. A careful history gives credit to all three stages rather than pretending the algorithm appeared once.
Why does ImageNet matter to language models when it’s a set of photographs?
Because ImageNet was the demonstration that made a much larger community take deep neural networks, large datasets and GPU training seriously. The architecture used for language is different, but the 2012 result helped move deep learning from a specialist research programme to the centre of industrial AI investment.
What is the single most important paper on this list?
If forced to choose, the 2017 Transformer paper is the obvious candidate because its architecture became the dominant basis of modern language models. But it was built from attention developed for translation, trained with backpropagation, accelerated by GPUs and TPUs, and fed tokenised data at enormous scale. Picking one paper rather misses the point of the page.
What did the frontier labs actually contribute?
A great deal, but mostly at a different layer: scale; the engineering and infrastructure required to make scale work; systematic empirical scaling; post-training and alignment; multimodal systems; inference and serving; product interfaces; and the willingness to spend extraordinary amounts on compute and talent. The labs added a great deal; they are also the latest builders on a much older foundation.
How long have the people around today’s frontier labs been working on this?
Much longer than the current companies have existed. DeepMind was founded in 2010, OpenAI in 2015 and Anthropic in 2021, while people connected to today’s frontier appear on this timeline in 2012, 2009, 1997 and 1986. The organisations are young. The intellectual careers behind them are not.
I’m new to AI. Where on this timeline should I start?
Read 1986, 2012 and 2017 first: backpropagation, AlexNet and the Transformer. Between them they explain how deep networks learn, why the field suddenly took off, and what architecture sits under the modern language-model era. Then read 2014 attention, 2016 subword tokenisation and 2022 InstructGPT to see how the chat model itself became possible. After that, go back and read the stalls, because they explain why the whole thing took eighty years.
Is this list complete?
No. The main timeline follows one deliberately narrow path from the artificial neuron to today’s conversational language models. Computer vision, speech, symbolic AI and reinforcement learning beyond games each get a paragraph rather than a full treatment, and plenty of researchers could make a good case for another milestone replacing one of mine. The cloud, the data centres, networking, virtualisation and containers that every model on this page was trained in and served from, is left out on purpose and needs its own history. A complete history would be a book, not a web page.
0 Comments