The Evolution of LLMs, Explained Through What We Train Them to Do

TL;DR: Every leap in LLMs, from GPT-1 to today's agents, came from changing one thing: what the model is trained to do. Next-token prediction gave models knowledge. Post-training made them assistants. Reinforcement learning made them reason. Tools made them agents. Underneath it all, the real engine is data designed to teach the right objective. Every few weeks, a new LLM launches and the feed calls it a revolution. Taken one at a time, the launches look like noise. There is a simpler way to read the story. Each big step answered one question: what should we train the model to do? First the answer was "predict the next word." Then it became "be a helpful assistant," then "think before you answer," and now "take actions until the job is done." Each step kept everything before it and added one idea on top. This mental model helps me make sense of the field, and it explains both how we got here and where things are going. Act 1: Predict the next token (2018 to 2020) GPT-1 (2018): the objective works. GPT-1 was a 117M-parameter transformer trained on one task: predict the next token in about 7,000 books. To do that well, it had to pick up grammar, facts, and meaning on its own. Fine-tuned on small labeled datasets, it beat purpose-built models on 9 of 12 benchmarks. Next-token prediction turned out to be a way to learn language itself. GPT-2 (2019): scale, and tasks start to appear. GPT-2 grew to 1.5B parameters trained on 40 GB of web text. Without fine-tuning, it began to summarize, translate, and answer questions zero-shot. The results were rough, but they hinted that scale alone unlocks new abilities. GPT-3 (2020): no more fine-tuning. At 175B parameters, GPT-3 could learn a task from a description and a few examples in the prompt, with no weight updates. Using a language model no longer required a labeled dataset, just a good prompt. Scaling laws showed that bigger models would keep improving predictably. The catch Next-token prediction teaches a model what text looks like, not how it should respond. A pretrained model: continues instead of answering. Ask "How do I bake bread?" and it may write five more questions, because that's what often follows a question online. doesn't know when to stay quiet. If it knows how to do something dangerous, nothing in its objective tells it not to say so. mirrors the internet, made-up facts and toxic text included. The capability was there. It just wasn't pointed anywhere useful. Act 2: Be a helpful assistant (2022) The fix was a second training phase: post-training. OpenAI's InstructGPT recipe had three steps: Humans write good example answers, and the model learns to imitate them. Humans rank the model's answers, and a reward model learns their preferences. The LLM is optimized against that reward model with reinforcement learning from human feedback (RLHF). The effect was dramatic. Raters preferred a 1.3B InstructGPT over the 175B GPT-3. The right objective beat 100x more parameters. In November 2022, OpenAI applied this recipe to a GPT-3.5 model tuned for dialogue and released ChatGPT. It answered instead of rambling and declined some harmful requests, and it reached an estimated 100 million users in about two months. From here on, the big advances happened after pretraining. Act 3: Think before you answer (2021 to 2025) Math and code came first. From 2021 to 2024, labs pushed hard on both, from Codex to trained math verifiers to step-by-step math verification. The two domains share one useful property: the answer can be checked automatically. A math answer matches, or it doesn't; code passes its tests, or it doesn't. That gives a cheap, reliable reward for reinforcement learning, with no human rater needed. This recipe is now known as reinforcement learning with verifiable rewards (RLVR), a term coined by AI2's Tülu 3 team in 2024. Then reasoning models. OpenAI's o1 (September 2024) was trained with large-scale reinforcement learning to write long chains of thought: trying approaches, catching errors, and backtracking. Each response is split into two: Hidden reasoning: long, messy working notes. Visible output: the clean answer the user reads. Technically it's one long answer, but nobody wants to read pages of scratch work every time. This also opened a new way to scale: spend more compute at answer time, not just at training time. In January 2025, DeepSeek-R1 showed in the open that pure reinforcement learning with simple rule-checked rewards (its R1-Zero variant) is enough for long reasoning to emerge. Act 4: Act until the job is done (2023 to 2026) A reasoning model still thinks only in text. It can't look anything up, run its code, or check its work. The next step moved reasoning from text space into action space: think, take an action (search, read a file, run a command), observe the result, and repeat until the task is done. The idea traces back to ReAct (2022). Function calling made it practical in 2023, and the Model Context Protocol (late 2024) gave models a standard way to plug into tools and data. By 2025, models like OpenAI's o3 and Anthropic's Claude 4 could call tools in the middle of their hidden reasoning. Agents then took over one domain after another: Search (December 2024 to February 2025). Google launched Gemini Deep Research first, followed by OpenAI and Perplexity. These agents run dozens of searches and return a cited report minutes later. Search came first because its actions are simple and its output is easy to check. Coding (2025). Claude Code (preview in February, generally available in May), OpenAI's relaunched Codex (May), and Cursor's agent mode read whole codebases, edit many files, run tests, and fix their own errors. Everything (2025 to 2026). A good coding agent turns out to be a general agent. OpenAI shipped ChatGPT agent in 2025. Anthropic launched Claude Cowork in January 2026 after seeing developers use Claude Code for almost everything. Agents now organize files, build spreadsheets, and draft reports. The deliverable is no longer an answer. It's finished work. Where we are now Progress hasn't plateaued: Agents got much better at real tasks. Stanford's 2026 AI Index reports success on Terminal-Bench rising from 20% in 2025 to 77.3% in 2026. Task length keeps doubling. METR estimates that the length of tasks agents can finish has been doubling roughly every four months since 2023. The strongest models now finish tasks that take human experts many hours, at 50% reliability. The frontier is still jagged, though. Models that beat PhD-level science questions can still fail at reading an analog clock. Why this happened in under a decade From GPT-1 to general agents took eight years. Three ingredients made that possible. 1. Models big enough to hold the patterns. A model can only learn what it has the capacity to represent, so scale matters. Getting to hundreds of billions of parameters took years of engineering, and much of it rests on simple ideas that keep information flowing through the network. One example: Residual connections (2015) let signals and gradients skip past layers, so very deep networks stopped breaking down during training. Normalization layers, attention, and better optimizers played similar roles. The broader lesson: as long as information flows cleanly through the network, it can learn. 2. Data that teaches the skill. With enough capacity, a model can learn almost any pattern present in its data. The flip side is that a skill with no data behind it won't appear. Much of the progress since 2018 is really progress in building datasets for new kinds of tasks. 3. The right objective, which is mostly data too. The objective decides what the model learns from its data, and each era changed it: Each era changed the training objective and kept everything before it. But objectives and data can't be separated. In supervised fine-tuning, the objective is the quality of the written solutions. In reinforcement learning, it is the reward; for tasks without one checkable answer, that is often a rubric that spells out what a good answer looks like. Both are data that someone had to create. So the real engine of progress is designing data that encodes the right objective. Nothing was thrown away along the way. Every agent today is still, at its core, a next-token predictor. What's next If data and objectives are the engine, the next step is clear: keep adding new kinds of tasks. Each new dataset type widens what models can do. Models also generalize, so skills learned on one task type carry over to related ones, and each addition lifts the rest. Plenty of problems remain open. Agents need reliability on long tasks: at 50% success an agent is a demo, but at 95% it's a coworker. Fuzzy work like writing and strategy needs good rubrics. And agents still start every task fresh, so models that remember and learn on the job would be the next big step. In 2018, GPT-1 needed fine-tuning for every new task. Eight years later, models built on the same core idea open files, run code, and hand back finished work. The next chapter will be decided less by bigger models and more by better objectives: what we reward agents for, and how we check their work. Which step mattered most: scale, RLHF, reasoning, or agents? Tell me in the comments.