Add an action input to a video model and it is steerable in principle. From Genie to Genie 3: how to tell conditioning from control, what scaling bought, and why fitting physics is not the same as having the rule.
If you have watched any generated video lately, you'll have seen somebody walk through a scene with the arrow keys. Until 2024 a video model was something you watched. It chose its own future and you had no say. One extra input changed that.
Take a model that predicts the next frame. Now tell it, along with the frames, which key is being held. From the same opening it can give you one continuation for left and another for right. Have a go with the figure below, and watch who picks each frame.
Press Next frame. You have no say in what arrives.
frames so far
2 of 8
chosen by you
0 of 0
the extra input
none
Jake Bruce and colleagues at Google DeepMind made that move with Genie in 2024. Their paper was called Genie: Generative Interactive Environments. Genie was trained on footage of 2D platform games, Mario and its descendants, and it turned them into worlds you could steer. Nobody had labelled the keypresses.
Instead, the model had to explain each change between frames with one of a handful of codes. The codes it settled on behaved like a controller. Genie also married two families that had grown up apart: world models built for agents (chapters 5 and 6) and video models built for audiences.
Twelve changes and no labels. The model has to file each one under a code.
pairs
12
codes in use
0 of 8
keypress labels given
0
A video model with an action input is somewhere you can be, and that is new. Whether it is a simulator, something that obeys your actions and holds the rules of the world, has to be measured, and the announcements tend to skip that.
An action input is not control
A model can receive an action and still ignore it, or respond only weakly. It can also let the action change how things look without changing the path underneath. Think of a steering wheel that turns without being connected to anything.
So control has to be measured. Hold the start fixed, vary only the command, and check whether the futures split apart as asked. I call it the steering-wheel test, and the figure below is that test as a picture.
Action input
present in every row
Left-right separation
107 px
Control test
The command changes the video, but weakly.
As you can see, the input can be there the whole time and mean very little. That failure is the one I'd expect. A model trained on footage is rewarded for frames that look right. If the footage mostly goes straight, going straight is a good bet whatever the key says.
Good video hides weak control. If three beautiful continuations all go straight, you have been shown video. A planner that asked for three counterfactuals, three what-ifs from one start, has been handed one of them three times.
From watching to walking in
Scaling did more than most people expected, including most of the people building it. Let's take the two systems that mark the change.
GameNGen came from Dani Valevski and colleagues at Google Research in 2024. Their paper was called Diffusion Models Are Real-Time Game Engines. A diffusion model makes a picture by starting from noise and cleaning it up step by step. GameNGen used one to run DOOM, the first-person shooter from the early nineties.
Start from noise. Each step cleans it a little, guided by the frames before and the key.
steps
0 of 12
time per frame
72 ms
frames a second
14
playable
about 20 a second, the playable line, illustrative
It made each frame from the earlier frames and the player's inputs, fast enough to play. People shown short clips struggled to tell it from the real game, its authors reported.
Genie 3, from Google DeepMind a year later, is where the numbers stopped looking like a demo. It is reported to make interactive video at 720p and 24 frames a second.
It is said to hold a scene together for several minutes and to take instructions part-way through: change the weather, add a flock of birds. Walk away from something, walk back, and it is roughly as you left it. Have a look at the clip below and watch for exactly that.
The public interface offers no stored 3D layout of the scene, and no list of its objects, that you could inspect. Yet DeepMind says Genie 3 can look back over its own path and keep visual information for about a minute. Whatever state supports that stays hidden inside the generator.
In under two years the field went from steering a blurry platformer to walking around a photoreal scene that remembered where you had been. Scaling did not give you anything you could open up and read.
Emergent physics, and where the footage runs out
You will see the claim made about all of them. OpenAI's 2024 report, Video generation models as world simulators, put emergent physics into common use. It was the write-up for Sora, OpenAI's own video generator. The title is the part that spread, and the claim in it bundles an observation with a conclusion.
The observation is real. Train on enough video and the outputs start to respect regularities nobody put in. Dropped objects speed up as they fall, water pools rather than piling up, and hidden objects come back into view when the thing in front moves. Nobody supplied an equation; those regularities are in the training footage, and predicting it well means copying them.
No equation anywhere. Only footage, and a drop it has not seen.
equations supplied
0
gap between frames
····
matches the footage
not yet
The conclusion is where the trouble starts. "It learned physics" says the model picked up the rule, and that is testable. The test has never been whether the outputs look right on the footage it was trained on.
Bingyi Kang and colleagues ran one in 2024, in a paper called How Far is Video Generation from World Model: A Physical Law Perspective. They trained video models on simple simulated scenes of balls moving and colliding, then asked for speeds the footage never showed. Figure 8.7 runs that test twice: once on speeds the footage covered, and once on speeds past them.
the speeds the footage covered
One model and a bench of scenes. Score it.
scenes on the bench
8
passed
0 of 8
the model
unchanged
Everything inside the band fits with having learned physics. It fits just as well with having learned the answers to the questions it was asked, and Kang's group concluded their models were doing the second: matching each new scene to the training scenes it most resembled and copying what happened there. Nobody tests outside the band, because that is where the footage runs out. I call that the band problem.
So the defensible claim is narrower. These systems copy a great many physical regularities across the situations they were trained on. Whether that is the same as having the rule, no demo you have seen can settle, because every one of them was inside the band.
What they are for
None of that argues against using them. Answering "it has not learned physics" with "so it is a toy" is the mirror-image error, and I see it nearly as often.
For training other systems a generated world is useful. It avoids contact with the real environment and costs GPU time instead. It resets at once, and it can produce a thousand versions of a scene that would take a week to stage. Press the buttons in the figure below a few times and you'll see what I mean.
a real stage
ready
versions so far
a generated world
ready
versions so far
Same scene on both sides. Press Another.
stage · versions
0
stage · time spent
0 days
stage · real contacts
0
stage · status
ready
generated · versions
0
generated · time spent
0 frames of GPU time
generated · real contacts
0
generated · status
ready
NVIDIA, whose chips train most of these models, built Cosmos for this. Cosmos is a set of video generators. The 2025 paper is called Cosmos World Foundation Model Platform for Physical AI. It makes physically-aware video as training data for robots and vehicles, where real data is dearest.
NVIDIA sells Cosmos beside its explicit, hand-built simulators as one platform, and has not declared either the replacement for the other. I'd take that as a hint about where things stand.
For being a world you can move around in, they are already good. No other approach lets you walk into the picture. I think that is worth more than the sceptics admit and less than the launches imply.
For being the model an agent plans inside, raw video generators are an awkward shape. A planner may need to score a hundred action sequences, and rendering each one is expensive. Systems can instead distil a learned latent, a short internal code that stands in for the frames, and plan in that.
10 sequences and the pixel bill is most of the chart. The latent planner paid a fraction and drew one.
Sequences
10
Frames rendered, pixels
10 × 8 = 80
Codes scored, latent
10 × 8 = 80
Frames drawn, latent
8
Cost, pixels
3,200 units
Cost, latent
400 units
Where the seams are
There are three seams where a demo can mislead you.
Consistency is learned, not enforced. Nothing stops the room from changing, and usually it does not. But usually is a statistical property of a trained system, and a guarantee is a property of an engine.
Nothing here is holding the door in place. Press and see how often it comes back anyway.
runs
0
held
0 of 0
guarantee
no, this is a rate
The interesting moments are the vague ones. A model trained on pixels is pushed towards the average of the possible futures wherever the future is open. We saw in chapter 7 what that average looks like: a picture of something that cannot happen.
The future is most open when something is about to happen. So the steering-wheel test is hardest to pass exactly where passing it matters. Step along the clip in the figure below and you'll see why.
every moment, both bars
It obeys here. Both commands end up in the same corridor, so obeying cost it nothing.
moment
1 of 6
at stake here
almost nothing
obeyed
yes, and it did not matter
The scores do not compose. Benchmarks measure visual quality, geometry, action fidelity and persistence one at a time. There is no accepted rule for combining them into one ranking.
So I will not tell you whether Genie 3 beats GameNGen. A claim about improvement belongs to one particular test. Click through the measures below and watch the winner change.
Frame quality, which is what demos are made of and what a per-pixel score rewards.
Best by this measure
System A
Best overall
There is no agreed way to combine these, so the question does not currently have an answer.
So, are video models world simulators? My answer is that they are somewhere you can be, and that is new and real. Whether one obeys you, and whether it holds the rules rather than the footage, are two separate measurements, and the demo you saw did not make either of them. Ask for the steering-wheel test, and ask what was outside the band.
Hopefully this chapter helped. There is a short quiz below if you want to check the ideas stuck. If I have got something wrong, or you know a better source than the ones I used, the About page has a corrections section and I would be glad to hear from you.
Try it
What change makes a video model capable of being steered, without proving that it will obey?
Pick one
1What change makes a video model capable of being steered, without proving that it will obey?
- a. Higher resolution
- b. A longer context window
- c. More parameters
- d. Conditioning each generated frame on an action as well as on the frames before it
Answer d. Conditioning each generated frame on an action as well as on the frames before it. Action conditioning supplies the interface. Controllability still has to be tested by varying the action from the same start and measuring whether the requested futures separate.
2A generated room stays as you left it when you turn back. What can you conclude about its memory?
- a. Some internal state supports persistence, but the behaviour does not reveal whether it is a scene graph, context or another mechanism
- b. It must use a separate object database
- c. It must contain exportable geometry
- d. It has no memory of any kind
Answer a. Some internal state supports persistence, but the behaviour does not reveal whether it is a scene graph, context or another mechanism. Behaviour establishes persistence, not implementation. Genie 3 is reported to refer back to its trajectory, while exposing no persistent geometry or object store to inspect.
3Dropped objects in generated video accelerate downwards, and nobody supplied gravity. What does that establish?
- a. Nothing at all
- b. The model has learned the law of gravity
- c. That the regularity is in the training footage, and reproducing it is required to predict the footage well
- d. The model contains a physics engine
Answer c. That the regularity is in the training footage, and reproducing it is required to predict the footage well. The observation is real. The question is whether reproducing a regularity across the range you trained on amounts to having the rule, and the observation alone does not settle it.
4In Figure 8.7, the model is within a few per cent across the band it was trained on. Why is that not evidence it has the rule?
- a. The measurement is unreliable
- b. Matching inside the band is equally consistent with having learned the answers to the questions it was asked
- c. A few per cent is too large an error
- d. The band was too narrow
Answer b. Matching inside the band is equally consistent with having learned the answers to the questions it was asked. Having the rule and having a fit through the sampled region only come apart outside the band, and outside is where nobody tests because there is no footage to compare against.
5Why is a generated video world the wrong shape for an agent to plan inside?
- a. It cannot be conditioned on actions
- b. It has no reward signal
- c. It is not accurate enough
- d. Planning means running the model many times over, and a frame is the opposite of a compact state
Answer d. Planning means running the model many times over, and a frame is the opposite of a compact state. A planner needs to try many action sequences and score them cheaply. These models are expensive to run and produce frames rather than something small to reason over.
6What are generated worlds unambiguously good for right now?
- a. Producing training data and being a place you can move around in
- b. Long-horizon planning
- c. Compressing video
- d. Replacing physics engines
Answer a. Producing training data and being a place you can move around in. They avoid real-world contact, reset instantly and produce broad variation, while still costing compute. No other approach in this course gives you an actual picture to walk into.
7Why does a wrong answer from one of these systems feel more convincing than a wrong number?
- a. It usually is not wrong
- b. The errors are smaller
- c. It arrives as something that looks like a photograph
- d. The models are better calibrated
Answer c. It arrives as something that looks like a photograph. The extrapolation problem here is the ordinary problem with any fitted function. What is different is how persuasive the output is when it fails.
8A system improves its geometry score but loses action fidelity. Is it better overall?
- a. Yes, geometry is the only objective measure
- b. Not without a declared use case or rule for combining the dimensions
- c. No, action fidelity always dominates
- d. Yes, any benchmark gain establishes overall progress
Answer b. Not without a declared use case or rule for combining the dimensions. Benchmarks now measure several useful dimensions. The unresolved part is how to combine them when systems and applications value different contracts.
Sources
- Genie: Generative Interactive EnvironmentsBruce et al., 2024Latent actions learned from unlabelled video, which is the move that turns a video model into somewhere you can be.
- Genie 3Google DeepMind, 2025The reported numbers this chapter quotes: 720p, 24 frames a second, minutes of coherence. A lab report rather than a result anyone outside has repeated.
- Diffusion Models Are Real-Time Game Engines (GameNGen)Valevski et al., 2024DOOM generated frame by frame from previous frames and inputs, fast enough to play. The clearest demonstration that this is playable rather than merely watchable.
- How Far is Video Generation from World Model: A Physical Law PerspectiveKang et al., 2024The measured version of Figure 8.7. Video models match physical laws within the distribution they were trained on, and do not extrapolate outside it.
- Video generation models as world simulatorsOpenAI, 2024The report that made the emergent-physics claim a mainstream one. Worth reading for exactly what it does and does not assert.
- Cosmos World Foundation Model Platform for Physical AINVIDIA, 2025Generated video built as training data for robots and vehicles, which is the use these systems are unambiguously good for.
- CosmosNVIDIAThe product framing, and a useful boundary case: generative video beside explicit simulation, sold as one platform.
This chapter is free and open source. A star helps the next person find it.
Star on GitHub22