A free interactive primer
World Models101
Since 2018, “world model” has meant at least five different machines, and the people saying it rarely tell you which. This primer pulls the five apart, then shows how each one works and where it came from.
Five things people mean
If you are not sure which kind you have been reading about, start here.
Ordered by what they predict
More concrete ← what gets predicted → more abstract
Not a system you run
1 of 5
A generator of observations, usually conditioned on your actions. You press a key, it produces the next frames; Sora is the same contract without the key. Persistence, where it exists, is learned through generation rather than guaranteed by explicit geometric state.
How to tell
Ask what holds the room together. If the answer is that the model learned to keep producing it, rather than that there is geometry, you are looking at a renderer.
- Who talks this way
- Generative video labs
- Covered in
- Chapter 08
The premise
Ask four labs what a world model is and you get four different machines that do not fit together. Each has its own history. Genie, from Google DeepMind, generates video you steer with the arrow keys. Marble, from World Labs, builds 3D geometry a robot can be trained inside. Dreamer, from Danijar Hafner's group, learns a small simulator and plans against it. V-JEPA, from Meta, predicts embeddings and throws the prediction away.
A fifth camp is not describing a machine at all. When people argue about whether a language model “has a world model”, they mean structure found inside a network that was trained for something else. The evidence comes from interpretability work, not from anything you could run. That is why two people can agree on every fact and still argue. One is asking whether a system can simulate. The other is asking whether a network contains something.
Under the naming mess is a real and old subject with a clear line of descent. It starts in 1960 with the Kalman filter, the maths that lets a radar track a plane from noisy blips. It passes through 1990, when Jürgen Schmidhuber and Richard Sutton first put a learned model of the world next to something that used it to act. It arrives at the four machines above.
Contents
9 chapters
- 01
What Is a World Model?
The phrase has covered at least five different machines since 2018, and the people using it rarely say which. A field guide to the five, where each came from, and the one test that separates them.
The map: five definitions of the term, and the question that tells them apart.
Read → - 02
How Do World Models Work?
A model you can run forward lets you try an action before paying for it. That is the oldest idea in the field and still the best one. The trouble is that a search good enough to find the best plan is also good enough to find the places where the model is wrong. Nobody has engineered that away.
A gap in traffic, two cars and a wall, and a planner that gets worse the harder it searches.
Read → - 03
Why Is Prediction the Same as Learning?
Ask something to predict what comes next and it has no choice but to build whatever the next moment depends on. Jeffrey Elman showed it with words in 1990. Claude Shannon had already shown that the same loop, read the other way, is compression.
One frame, then two: watch the futures still open collapse as evidence arrives.
Read → - 04
What Is Latent Space?
A camera measures tens of thousands of numbers and the decision needs two or three. What happens at the squeeze between them, and why that narrow point sets the ceiling on everything downstream.
Drag a point through a two-number space and watch the room it decodes to.
Read → - 05
What Is a Dynamics Model?
In training, the model is handed the truth at every step. The moment you deploy it, it gets its own last answer instead. What carries the past forward, and why the headline accuracy number measures a job the model will never be asked to do.
One model, two ways of running it: corrected every step, or left to eat its own output.
Read → - 06
Can an AI Learn Inside Its Own World Model?
A month of robot time becomes a day if the practice happens inside the model. That has been the pitch since Dyna, and Dreamer made it work. What the exchange rate costs, and why the fix for an agent that exploits its own dream is to make the dream worse on purpose.
Turn the uncertainty in the dream up and down, and watch both scores fail at opposite ends.
Read → - 07
What Is JEPA, and Why Not Predict Pixels?
When a deterministic pixel predictor meets an open future, its best answer can be a picture of something that cannot happen. Yann LeCun's case against generation, what sampling fixes, what embeddings avoid, and what each still owes.
Two possible futures, and the smear that scores better than either of them.
Read → - 08
Are Video Models World Simulators?
Add an action input to a video model and it is steerable in principle. From Genie to Genie 3: how to tell conditioning from control, what scaling bought, and why fitting physics is not the same as having the rule.
One start, three futures, chosen by which key you hold.
Read → - 09
What Is Still Broken in World Models?
Scenes do not fail all at once, and different systems fail under different contracts. The closing argument: what to test, which benchmark claims compose, and which do not.
Drag out to a thousand steps and read which properties have already gone.
Read →
Who this is for
The engineer
You ship models but the RL-adjacent literature has always felt like a different country. You want a map of it.
The researcher next door
You work in another subfield and need to know what people mean when they say a video model understands physics.
The student
You have the maths and the time and want a path through the papers that is ordered rather than alphabetical.
The sceptic
You suspect a lot of this is overclaimed. Chapter 9 is for you, and you are probably partly right.
NiluK/worldmodels101
Everything here is open source
The prose, the figures and every interactive on this site are in one public repository. Read it, correct it, lift a component for your own writing. If it was useful, a star is the cheapest way to help somebody else find it.