Barlow Twins → VICReg → JEPA → DINO → SigReg: five answers from computer vision to the same problem, each implemented in our model and measured against the same control. Two helped. Three cost.
A system trained to predict in latent space has a trivial solution at hand: ignore the input and produce the same representation for everything. Then the prediction is perfect and useless. The talk calls this collapse, and describes it in energy terms: the energy function becomes flat, zero everywhere — the autoencoder that learns the identity, the JEPA that emits constants. Every joint-embedding self-supervised method is, before anything else, an answer to this.
There are two families of answers. Contrastive: generate points outside the data and push their energy up (CLIP is the big example). Regularized: limit the volume of space that can have low energy — if only a small volume fits, pushing the data down forces the rest to rise. The talk prefers the second, and the version of it that reduces to practice is information maximization: take the matrix of representations of a batch (one row per sample, one column per dimension) and make the columns differ from each other. Methods that do this are "contrastive per dimension" — and they are all the ones in this chapter. There has never been a negative pair in this project.
Each link in the lineage chooses what to regularize and how. What changes is the statistic being forced: cross-correlation, variance and covariance, a target with stop-gradient, a distribution over prototypes, or full Gaussianity. Our test is the same for all of them: the piece goes into the 7.4 M mini on top of the control (cosine 1.0), same frozen data, 700 steps, seed 1337, best checkpoint. The question is not "which is best" — it is how much each property costs, and whether any comes for free.
| Link | What it forces | On what, here | Flag |
|---|---|---|---|
| Cosine (the control) | mean cosine between random pairs → 0 | the embedding table | --reg-cos |
| Barlow Twins (2021) | cross-correlation matrix between two views → identity: diagonal 1, off-diagonal 0 | rudder prediction × target (the "two views" are the predicted and the actual) | --barlow |
| VICReg (2021) | variance of each dimension above a threshold + covariance between dimensions → 0 | the embedding table | --vicreg |
| JEPA (2022) | predict the target's representation, with the target without gradient (stop-grad) or coming from a teacher | the deep rudder: Pred( | --latent --latent-target oculto |
| DINO (2021) | self-distillation: teacher = exponential moving average of the student; target = distribution over | the same deep rudder, with an EMA teacher and 1,024 prototypes | --dino |
| SigReg (LeJEPA, 2025) | the distribution of the representations → isotropic Gaussian, via many 1-D projections compared to the Gaussian (Epps-Pulley) | the table (--sigreg) or the states (--sigreg-z, chapter 05) | --sigreg |
Two details the table hides. SigReg is the only one that forces the whole distribution, and not a second-order statistic: that is why it sees rank collapse that the cosine does not see (chapter 03: rank 8 of 256, cosine reports +0.003, SigReg reports 11× the floor). And DINO is the only one with a teacher: the target is not the state itself with stop-gradient, it is the state of a slow copy of the model — the BYOL recipe, which the talk describes as "the encoder on the right uses a moving average of the weights of the one on the left".
| Run | validation | vs control | aniso | decorr | rank | rudder |
|---|---|---|---|---|---|---|
no regularizer (moderno) | 5.2616 | −0.17% | 0.378 | — | 232.4 | — |
cosine (reg1, control) | 5.2706 | — | 0.049 | — | 244.8 | — |
SigReg on the table (sig1) | 5.3503 | +1.51% | 0.054 | 0.068 | 245.0 | — |
VICReg + cosine (reg1-vic1) | 5.4226 | +2.88% | 0.029 | 0.017 | 253.0 | — |
JEPA: deep rudder, stop-grad (reg1-lat1o) | 5.2233 | −0.90% | 0.052 | 0.061 | 244.6 | 0.716 |
Barlow Twins on the deep rudder (reg1-lat1o-bar0.5) | 5.2442 | −0.50% | 0.054 | 0.061 | 244.7 | 0.728 |
DINO: EMA teacher on the deep rudder (reg1-lat1o-dino1) | 5.3804 | +2.08% | 0.052 | 0.062 | 245.0 | 0.703 |
SigReg + deep rudder (sig1-lat1o) | 5.3856 | +2.18% | 0.059 | 0.070 | 244.8 | 0.719 |
reg1-lat1o) was added after measuring sig1 — testing the newest piece on the weakest base would confound the two.SigReg on the table bought nothing. VICReg bought the best geometry and paid 2.88% for it. DINO's EMA teacher cost 3 percentage points relative to stop-gradient. None of the three says the method is bad — it says that, in this model, on this budget and with this seed, the statistic they force is not the one that was missing. SigReg gets its proper chance in chapter 05: on the states of the world model (--sigreg-z), where collapse is real and the cosine on the table does not reach.
The reading in §3 draws the recipe of the world model (chapter 05): keep the cosine on the table (cheap, solves the cone), predict the next state in the latent (the only piece that improved the generator), and put the anti-collapse where collapse happens — on the states, with SigReg, without an EMA teacher. It is exactly the LeWorldModel recipe, arrived at by measurement and not by citation: the wm1-sz1 run tests it, and the wm1det control (stop-grad, without SigReg-z) says whether SigReg on the states plays the role the recipe gives it.
Five remedies for the same fever. Four attack the thermometer — they make the embedding table look healthier on every instrument. One attacks the cause: it forces the model to know where it is going to end up. The table in §3 is what happens when you measure the patient instead of the thermometer.
Chapter 05 takes the only piece that improved the generator and turns it into the one that imagines: the world model.