The Idea
11 min read · interactive
The oldest of the five definitions, and the one the term was coined for. What a learned simulator is, what having one buys you, and the way it fails that no amount of engineering has removed.
A professional tennis serve arrives in about half a second.
Getting a signal from your retina to your visual cortex takes somewhere around sixty to a hundred milliseconds. Getting a large muscle group to actually move takes another hundred and fifty. Add the swing and the returner is over budget before they have decided anything at all.
So they are not reacting to the ball. They cannot be. They are reacting to a prediction of it, produced by something that took the toss, the racket angle and the first few frames of flight, and ran them forward.
That is the Dynamics Model, the definition Chapter 1 said the term was coined for. This chapter is about what it is, what having one buys you, and the way it fails that no amount of engineering has removed.
Cached answers
Two identical cars approach two identical walls.
The first brakes when the wall is closer than some fixed distance. Somebody tuned that distance once, it worked, and it shipped. The second carries a model of its own braking, rolls it forward every tick, and brakes the moment its imagined stopping point ends up inside the wall.
Below about forty they behave identically. That is not a warm-up, it is the point. A reflex is cheap, fast, and correct across the whole range it was tuned in, which is why evolution and engineering both spend most of their time building them.
Push past the tuned range and the reflex does not degrade gracefully. It fails at a threshold, completely, and it fails with no internal signal that anything has changed. The fixed trigger distance encodes an assumption about speed, and nothing in the controller knows it is making one.
The model car slows earlier because its stopping distance grew and it can see that it grew.
What it buys you
Three things, and they are all the same thing wearing different clothes: you can ask questions about situations that have not happened.
You can try an action before paying for it. Chapter 1 defined this category by whether you can search inside it. Here is what that means in practice.
Closest so far, still short.
Considered
1
Would have hit the wall
0 / 1
Best distance from goal
136
With one sample it is guessing. With two hundred it reliably finds the way round. Nothing about the world changed between those two states; the only thing that changed is how much thinking happened before acting, and a model is what converts thinking into a better action.
You can practise where mistakes are free. Reality is a slow and expensive teacher. Every trial costs wall-clock time, and some trials cost a drone. A model runs faster than real time and breaks nothing, so once you have one you can generate as much experience as you have compute for. This is the largest practical argument for model-based methods and the subject of Chapter 6.
You can ask what would have happened. Given a model, what if I had braked earlier is answerable. Without one it is not, because the world only ever runs the experiment once. Credit assignment, regret, and most of what we informally call reasoning depend on being able to ask it.
Where it breaks
Everything above assumed the model is right. It is not, and the way it is wrong is worse than being noisy.
A learned model is wrong in specific, structured places: the parts of the world it saw least during training. That would be tolerable if you sampled plans at random. But you do not. You search for the plan the model scores highest, and a search that is good at its job will find exactly the places where the model is most optimistic, which are disproportionately the places where it is most wrong.
The optimiser is not fighting you. It is doing precisely what you asked.
Not enough search to find the flaw. It takes the long way round, and it works.
In the model
Reached the goal
In reality
Reached the goal
Read that figure twice, because it inverts something most engineering intuition takes for granted. A weak optimiser produced a plan that worked. A strong one produced a plan that was perfect in imagination and hit a wall. More search made the outcome worse, and it did so by getting better at its stated job.
This has a name, model exploitation, and it is why so much of the field's machinery exists. Ensembles, uncertainty penalties, pessimism, short rollouts and trust regions are all versions of one instruction: do not let the planner go where the model has not earned the right to be confident.
Planning is the process of taking a model as input and producing or improving a policy for interacting with the modelled world.
Which is exact, and contains the whole difficulty in the phrase the modelled world. The policy you get is good for the world in the model. Whether that is the world you are standing in is a separate question, and nothing inside the model can answer it.
What to carry into Chapter 3
- A reflex is a cached answer. A model recomputes when the question changes, and earns its cost only outside the range the cache was built for.
- The defining property is searchability, not fidelity. If you cannot roll it forward under actions nobody has taken, it is not doing this job.
- A model buys you three things that are one thing: trying actions before paying for them, practising where mistakes are free, and asking what would have happened.
- A better optimiser can produce a worse outcome, because it finds the places your model is confidently wrong. That is the shape of the problem, not a bug in your implementation.
Everything here assumed you already had a state to predict from. You do not. You have pixels.
Thanks for reading. Chapter 3 is about why predict the next thing, an objective that sounds far too simple to work, turns out to be enough to pull the structure of a world out of raw experience.
Try it
Below about forty on the braking demo, the reflex car and the model car behave identically. What does that tell you?
Pick one
1Below about forty on the braking demo, the reflex car and the model car behave identically. What does that tell you?
- a. The model is not working yet
- b. A reflex is correct across the range it was tuned in
- c. The model is too slow to matter at low speed
- d. The demo is not sensitive enough
Answer b. A reflex is correct across the range it was tuned in. A reflex is a cached answer, and inside its range the cache is right. That is why reflexes are everywhere and why most of the time you should not reach for a model at all.
2The reflex car fails past its tuned range. What is the worst part of how it fails?
- a. It fails gradually, so nobody notices
- b. It fails suddenly, with no internal signal that anything changed
- c. It fails only at very high speed
- d. It brakes too early instead of too late
Answer b. It fails suddenly, with no internal signal that anything changed. The fixed trigger distance encodes an assumption about speed, and nothing in the controller knows it is making one. The model car slows earlier because its stopping distance grew and it can see that it grew.
3Chapter 1 said this category is defined by searchability rather than fidelity. What does that rule out?
- a. A model that is fast but approximate
- b. A photorealistic model you cannot roll forward under hypothetical actions
- c. A model that works only in simulation
- d. A model trained on a small dataset
Answer b. A photorealistic model you cannot roll forward under hypothetical actions. Looking right is not the job. If you cannot ask it what happens under an action nobody has taken, you cannot search over actions, and searching over actions is the whole reason to carry it.
4In the planner, going from one sampled action sequence to two hundred produces a much better plan. What changed?
- a. The model got more accurate
- b. The world became easier
- c. Only the amount of thinking done before acting
- d. The goal moved closer
Answer c. Only the amount of thinking done before acting. Nothing about the world or the model changed between those two states. A model is what converts extra computation into a better action, which is a trade almost nothing else in control gives you.
5Which of these is NOT something having a model buys you?
- a. Trying an action before paying for it
- b. Practising where mistakes are free
- c. Asking what would have happened if you had acted differently
- d. A guarantee that the plan will work in the real world
Answer d. A guarantee that the plan will work in the real world. The first three are the same capability wearing different clothes: asking questions about situations that have not happened. The fourth is exactly what a model cannot give you, and the rest of the chapter is about why.
6In the exploitation demo, raising the search effort makes the real outcome worse. Why?
- a. The optimiser is buggy
- b. More samples add noise to the plan
- c. A better search finds where the model is most optimistic, which is often where it is most wrong
- d. The model degrades as it is queried more
Answer c. A better search finds where the model is most optimistic, which is often where it is most wrong. The optimiser is not fighting you; it is doing exactly what you asked. You told it to find the plan the model scores highest, and the model scores highest in the places it never saw during training.
7A learned model is wrong in a way that is worse than being noisy. What is the difference that matters?
- a. Its errors are larger than noise
- b. Its errors are structured, concentrated where it saw least during training
- c. Its errors grow over time while noise does not
- d. Noise can be averaged out and model error cannot
Answer b. Its errors are structured, concentrated where it saw least during training. Random error would be tolerable, because a random search would meet it at random. Structured error is dangerous precisely because a directed search seeks out the optimistic regions, and those are the same regions.
8Ensembles, uncertainty penalties, pessimism, short rollouts and trust regions are all versions of one instruction. Which?
- a. Make the model more accurate
- b. Collect more real data before planning
- c. Do not let the planner go where the model has not earned confidence
- d. Prefer model-free methods when in doubt
Answer c. Do not let the planner go where the model has not earned confidence. All of them constrain where the plan is allowed to look rather than trying to make the model globally right, which nobody knows how to do. Chapter 8 comes back to this as the field's central unfinished business.
Sources
- Dyna: an Integrated Architecture for Learning, Planning and ReactingSutton, 1991Planning defined as computation over a learned model, and the loop that interleaves it with real experience.
- World ModelsHa & Schmidhuber, 2018Encoder, dynamics, controller, with the policy trained almost entirely inside the model's rollouts.
- Learning Latent Dynamics for Planning from Pixels (PlaNet)Hafner et al., 2018Planning by search over action sequences in a learned latent space, which is Figure 2.2 in its real form.
- Dream to Control (Dreamer)Hafner et al., 2019Behaviour learned from long imagined rollouts rather than single-step lookahead.
- When to Trust Your Model: Model-Based Policy OptimizationJanner et al., 2019Short rollouts as a direct answer to exploitation: the model is trusted only as far as it has earned.
- Benchmarking Model-Based Reinforcement LearningWang et al., 2019Where model-based methods actually win and lose, including how often the planner exploits the model.
- MOPO: Model-based Offline Policy OptimizationYu et al., 2020Pessimism made explicit: penalise reward by model uncertainty so the planner stops being rewarded for going somewhere unknown.