The lineage of collapse: five answers from computer vision, each measured on text

A system trained to predict in latent space has a trivial solution: ignore the input and always return the same vector. Perfect and useless prediction. LeCun calls that collapse, and describes it in terms of energy: the function goes flat, zero everywhere. Computer vision gave five answers in sequence: Barlow Twins, VICReg, JEPA, DINO, SigReg. We implemented all five in our text model and measured against the same control. Two helped. Three cost. VICReg bought the best geometry in the table and paid 2.88% of quality for it. No negative result says the method is bad. It says that, in this model, at this budget, the statistic it enforces is not the one that was missing.

The three points

1. Two families of answer: contrastive (push fake points up) and regularised (limit the volume of low energy). LeCun prefers the second; here there was never a negative pair.
2. Each link, measured: what it enforces, what it bought in geometry, what it cost the generator.
3. Three negative results stated with the same prominence. SigReg bought nothing on the table, but has its rightful chance on the world model's states.

From the chapter, the beginning

Barlow Twins → VICReg → JEPA → DINO → SigReg: five answers from computer vision to the same problem, each implemented in our model and measured against the same control. Two helped. Three cost.

A system trained to predict in latent space has a trivial solution at hand: ignore the input and produce the same representation for everything. Then the prediction is perfect and useless. The talk calls this collapse, and describes it in energy terms: the energy function becomes flat, zero everywhere — the autoencoder that learns the identity, the JEPA that emits constants. Every joint-embedding self-supervised method is, before anything else, an answer to this.

There are two families of answers. Contrastive: generate points outside the data and push their energy up (CLIP is the big example). Regularized: limit the volume of space that can have low energy — if only a small volume fits, pushing the data down forces the rest to rise. The talk prefers the second, and the version of it that reduces to practice is information maximization: take the matrix of representations of a batch (one row per sample, one column per dimension) and make the columns differ from each other. Methods that do this are "contrastive per dimension" — and they are all the ones in this chapter. There has never been a negative pair in this project.

Each link in the lineage chooses what to regularize and how. What changes is the statistic being forced: cross-correlation, variance and covariance, a target with stop-gradient, a distribution over prototypes, or full Gaussianity. Our test is the same for all of them: the piece goes into the 7.4 M mini on top of the control (cosine 1.0), same frozen data, 700 steps, seed 1337, best checkpoint. The question is not "which is best" — it is how much each property costs, and whether any comes for free.