When a deterministic pixel predictor meets an open future, its best answer can be a picture of something that cannot happen. Yann LeCun's case against generation, what sampling fixes, what embeddings avoid, and what each still owes.
If you've followed the world model argument online, you'll have run into a slogan: stop predicting pixels. It comes from Yann LeCun. He was Meta's chief AI scientist at the time, and he is a Turing Award winner. He is also one of the people who made convolutional networks work, the kind of network that reads images.
In 2022 he published A Path Towards Autonomous Machine Intelligence. It is a position paper, meaning a case for how machines should learn rather than a result. One charge in it was aimed at the reflex then governing learned world models: when in doubt, predict the next picture.
His smaller charge starts with a camera pointed at a road. Ask what the next second looks like: a leaf at the edge of the frame will move, and which way is unknowable. It is also the least useful fact on offer, since nothing you might do about the road depends on it. A model asked to predict the picture has to predict that leaf anyway.
The leaf is made of pixels, and pixels are what the model is marked on. Multiply that by every leaf, every ripple, every flicker of light on wet tarmac. Most of the capacity has gone to things that are hard to predict and do not matter. That is the leaf problem, the smaller of the two charges.
what the model predicted
what happened next
where the error landed
each cell tinted by its squared error
pixels a driver cares about
the car, the road edge
10% of the error
15% of the frame
pixels no decision depends on
leaves, puddle, light
90% of the error
30% of the frame
Most of the error, and so most of the push on the weights, is on pixels no decision depends on.
shades, errors and shares are illustrative
second
1
share of the error on the car and road edge
10%
share on leaves and water
90%
JEPA is a family of models from Meta. It predicts a description of what it cannot see, instead of the pixels. An embedding, in everything that follows, is a short list of numbers standing in for a picture.
An average of two futures is neither
The larger charge is a car coming to a junction. It will turn left or right. You do not know which, and no amount of staring at the current frame will tell you.
Now let's ask a deterministic model, one that gives a single answer every time, for one next picture. Score it by squared pixel error, the gap at each pixel squared and added up. The best answer in the maths is the conditional average of the two turns. That is the left and right futures blended at their odds, rather than either one on its own.
one possible future (50%)
the best possible pixel prediction
the other one (50%)
Is the middle panel a thing that can happen?
No. It is both futures at once, and neither of them is that.
Does it score better than either real future?
Yes. Squared error over a set of outcomes is minimised by their average, so the optimum is a picture of nothing.
Have a look at the middle panel. It scores better than either real future. A model that predicted the left turn would be punished half the time. The ghost in the middle is punished a little all the time, and under squared error a little all the time wins.
So that training signal asks for the mean rather than for a plausible future. When the future is open, the mean is a smear that never occurs.
The ghost had an answer five years early
One complication, though. The pixel camp had dealt with the ghost by 2017, five years before the position paper. SV2P, short for Stochastic Variational Video Prediction, is Mohammad Babaeizadeh and colleagues' video model from that year, and Emily Denton and Rob Fergus built one like it. Both learn a spread of futures and draw one at random.
The trick is a hidden variable, a few numbers standing in for whatever the frame cannot tell you, such as which way the car turns. Draw the variable first, then the frame that goes with it. Each draw can be sharp even when the model has no idea which turn will happen.
Individual prediction
Blurred: it lands between outcomes.
Uncertainty
Collapsed into one average image.
That kills the blur objection, though not the rest of the case.
Sampling fixes the form of the output. It does not make useless texture cheap, make rare futures well estimated, or tell a decision maker which possible future matters. Stochastic generation keeps the pixel target, and embedding prediction changes it.
Predict the description instead
LeCun's proposal is to stop predicting the picture. Encode what you can see into a compact description (an embedding, a short list of numbers that stands in for something bigger). Similar things land near each other in that list. Encode the part you are predicting into one too.
Then train the model to predict the second description from the first. Nothing is ever decoded back into pixels. I-JEPA is the simplest example: a 2023 image model from Mahmoud Assran and colleagues in LeCun's group at Meta. Hide part of the picture and predict the embedding of the missing piece. What changes with the target is what counts as a bad prediction.
what happened
what the model said
both scales are illustrative
Every leaf is somewhere else. Nobody could have called where they went and nothing you would do about the road depends on it, and the pixel scale charges the most for it. The two scales are not measuring the same thing.
Change
the leaves
pixel error
70
description distance
4
V-JEPA 2, from Meta AI in 2025, does the same for video. It is trained first on plain video, with no actions. Then it is trained again on a robot's actions, for control.
Two things follow. The leaf problem goes away, because the description never had to record which way the leaf went. If a detail does not survive the encoding, nothing downstream is graded on it.
The ghost shrinks as well, because you are no longer averaging pictures. The descriptions of a car turning left and one turning right can share most of their entries. The part still open is either kept explicit or dropped as irrelevant. One caution: a space of descriptions can still hold two separate answers, and changing the representation does not get rid of uncertainty.
turns left
the average picture, which never happens
turns right
the lists are illustrative
Averaging two pictures gives a picture of neither.
entries the two futures share
7 of 8
entries still open
1 of 8, kept explicit
dropped as irrelevant
none
The bill for that
Predicting pixels has one large virtue, and the JEPA side gives it up. The target is fixed, and the model cannot make the next frame easier to predict. Predict an embedding and that stops being true.
The target now comes from the encoder, the network that turns pictures into descriptions, and that network is also being trained. It can get a perfect score while learning nothing, by making every description the same. The prediction is then always right, the loss goes to zero, and the descriptions say nothing about anything. That failure is called collapse.
how close the prediction is
0.00000
how much the representation still says
0.000
What you have got
A perfect score and an empty representation. Everything encodes to the same point.
Most of the difficulty in these methods is preventing that, and the fixes form a lineage. In 2020 the received wisdom for learning descriptions without labels was contrastive, meaning you learn by comparing pairs. Pull two views of the same image together. Push everything else apart, so the descriptions cannot all pile up in one place.
BYOL, short for Bootstrap Your Own Latent, is a 2020 method from Jean-Bastien Grill and colleagues at DeepMind. It dropped the pushing apart. Instead it keeps a second copy of the encoder that updates slowly (usually called a target encoder, it trails the one being trained), and it predicts that copy's output. The target cannot run away from the predictor.
By the standard account it should have collapsed. It did not, and that convinced people collapse could be avoided without pushing anything apart.
VICReg, a 2021 method from Adrien Bardes, Jean Ponce and LeCun, went the explicit route instead. The name spells out its three terms: variance, invariance, covariance. It penalises descriptions whose parts have collapsed or copied each other. That is Figure 7.6's safeguard done properly.
Pairs pulled together, everything else pushed apart. The cloud cannot pile up because every point is being shoved.
spread
1.00
steps taken
0
Randall Balestriero and colleagues wrote a 2023 survey, A Cookbook of Self-Supervised Learning. It is frank that much of the field is machinery for stopping the trivial solution from winning.
The loss is also no longer a number you can read. A pixel loss of zero means the model predicted the frame. An embedding loss of zero might mean it understands everything, or nothing at all. The number will not tell you which.
curves are illustrative
Both curves reach zero. Press Reveal.
pixel loss at the end
0
what that zero tells you
not yet revealed
embedding loss at the end
0
what that zero tells you
not yet revealed
So Quentin Garrido and colleagues tested what the embeddings could do. Their 2024 paper is Learning and Leveraging World Models in Visual Representation Learning. They found that the strength of the predictor decides what the encoder keeps. Give the predictor an easy job and the encoder throws away whatever the transformation changed. Make it work, and the encoder keeps enough to undo the change, which is more than the loss ever asked for.
What this does not settle
None of this makes generation a mistake. The version of the argument that says it does is the overclaim. Systems that predict pixels are the ones making worlds you can walk around in, and they work. Whatever they waste on leaves, they deliver a picture, and no embedding does that.
The awkward counter-example comes from LeCun's own lab. Kaiming He, best known for ResNet, the network design computer vision ran on for years, was at that lab in 2021. His paper that year was Masked Autoencoders Are Scalable Vision Learners. It hides most of an image and rebuilds its pixels, the very thing JEPA refuses to do.
Three quarters hidden. Press Rebuild and the decoder has to redraw it.
hidden
75% of the image
pixel loss
clusters the encoder separates
untested
The descriptions it learns are very good anyway. It came out the year before the position paper.
The version of the argument that holds up is narrower than the slogan, but it is right. Predicting appearances spends most of its capacity on things no decision depends on. When the future is open, a deterministic predictor is trained towards a picture of something that will not happen.
If what you want is a compact state to act on, that is a bad way to get one. If what you want is a picture, it is the only way anyone has. Most of the confusion in this corner of the field comes from comparing systems built for different goals.
Hopefully this chapter helped. There is a short quiz below if you want to check the argument stuck, including the part about what it does not settle. If I have got something wrong, or you know a better source than the ones I used, the About page has a corrections section. I would be glad to hear from you.
Try it
A deterministic predictor uses squared pixel error and a car turns left or right with equal probability. What is its optimum?
Pick one
1A deterministic predictor uses squared pixel error and a car turns left or right with equal probability. What is its optimum?
- a. The average of the two, which is a picture of neither
- b. Whichever was more common in training
- c. The left turn
- d. The right turn
Answer a. The average of the two, which is a picture of neither. Squared error over a set of outcomes is minimised by their mean. Committing to one turn is punished half the time; the smear is punished a little all the time, and that wins.
2A stochastic video model now draws sharp left and right turns instead of their blur. What has it fixed, and what has it not?
- a. It removed the need for a pixel target
- b. It proved embedding prediction is unnecessary
- c. It has learned which turn the real car will choose
- d. It fixed the impossible average but still has to represent probabilities and decision risk
Answer d. It fixed the impossible average but still has to represent probabilities and decision risk. Sampling is a real solution to blur: each draw can be plausible. It does not identify which draw reality will select or tell a planner how to value the distribution.
3A leaf at the edge of the frame moves unpredictably. Why does a pixel-predicting model spend capacity on it?
- a. Leaves are important for scene understanding
- b. The leaf is made of pixels, and pixels are what the model is marked on
- c. It cannot tell leaves from cars
- d. The encoder forces it to
Answer b. The leaf is made of pixels, and pixels are what the model is marked on. Nothing tells the objective which pixels matter. Hard-to-predict and irrelevant is the worst combination, and it is most of the frame.
4What does predicting an embedding rather than a picture do to the leaf problem?
- a. Nothing, the leaf is still in the input
- b. It makes the leaf easier to predict
- c. If a detail does not survive the encoding, nothing downstream is graded on it
- d. It moves the problem to the decoder
Answer c. If a detail does not survive the encoding, nothing downstream is graded on it. The description was never obliged to record which way the leaf went, so the model is not punished for failing to say.
5Predicting pixels has one large virtue that predicting embeddings gives up. What is it?
- a. The target is fixed: the model cannot make the next frame easier
- b. It needs less data
- c. It generalises better
- d. It is faster to compute
Answer a. The target is fixed: the model cannot make the next frame easier. The moment the target is produced by a network that is also being trained, the target can move, and there is a way to make it move somewhere very convenient.
6What is collapse?
- a. The model forgetting earlier training
- b. Gradients vanishing in deep layers
- c. The loss diverging to infinity
- d. Every input encoding to the same description, so the prediction is always right and the representation says nothing
Answer d. Every input encoding to the same description, so the prediction is always right and the representation says nothing. It is a perfect score obtained by learning nothing, and it is the cheapest solution available unless something is specifically stopping it.
7In Figure 7.6, the run with the safeguard has a worse loss. What does that tell you?
- a. The safeguard is badly tuned
- b. An embedding loss is no longer a number you can read off as quality
- c. The model needs more training
- d. The safeguard should be removed once training stabilises
Answer b. An embedding loss is no longer a number you can read off as quality. A pixel loss of zero means the frame was predicted. An embedding loss of zero might mean everything or nothing, and the number cannot tell you which.
8Does this argument show that generating pixels is a mistake?
- a. No, because collapse makes embeddings unusable
- b. Yes, it is strictly worse
- c. No. It is a bad way to get a compact state to act on, and the only way anyone has to get an actual picture
- d. Yes, except for very short videos
Answer c. No. It is a bad way to get a compact state to act on, and the only way anyone has to get an actual picture. The two goals were never the same goal, and most of the confusion here is people comparing systems built for different ones.
Sources
- Stochastic Variational Video Prediction (SV2P)Babaeizadeh et al., 2017The real counterargument to deterministic blur: learn a distribution and draw distinct plausible futures instead of returning their pixelwise mean.
- A Path Towards Autonomous Machine IntelligenceLeCun, 2022The position paper the whole argument comes from, including why predicting appearances is the wrong job for a system meant to act.
- I-JEPAAssran et al., 2023Hide part of an image and predict the embedding of the missing piece rather than redrawing it. The clean statement of the method.
- V-JEPA 2Meta AI, 2025The video version, pre-trained without actions and then post-trained for control, which is where the argument meets a robot.
- Bootstrap Your Own Latent (BYOL)Grill et al., 2020The slowly-updating target copy, and the result that made people believe you could avoid collapse without pushing things apart.
- VICRegBardes, Ponce & LeCun, 2021The explicit approach: penalise a representation whose components have collapsed or duplicated each other. Figure 7.6's safeguard, done properly.
- A Cookbook of Self-Supervised LearningBalestriero et al., 2023The survey that says how much of this field is machinery for stopping the trivial solution from winning.
- Masked Autoencoders Are Scalable Vision LearnersHe et al., 2021The counter-example worth holding on to: reconstruct the pixels of the masked part, and it works very well anyway.
- Learning and Leveraging World Models in Visual Representation LearningGarrido et al., 2024What the embedding-prediction objective turns out to have learned, tested rather than asserted.
This chapter is free and open source. A star helps the next person find it.
Star on GitHub21