The geometry of the embedding: training starts with a perfect space and ruins it

We tried semantic search with RoBERTa. Every similarity came out above 0.99. Sentences on opposite subjects looked identical. At the time we diagnosed it as anisotropy and moved on. Here the question became another: is that a RoBERTa defect, or does it happen to any model? We measured on ours. The space starts perfectly isotropic and training creates the cone. It is no defect of any specific model: it is what happens when you train embeddings with softmax and weight tying. RoPE, RMSNorm and SwiGLU change nothing. It has had a name in the literature since 2019, and four instruments to see it, none of which is enough alone.

The three points

1. Anisotropy, effective rank, decorrelation and probes: each instrument has a blind spot. One alone misleads.
2. The finding: the cone is created by training with softmax and weight tying. It has a name: representation degeneration problem (Gao et al., 2019).
3. The simple remedy is a cosine regulariser. It exists because a semantic search failed, and it is the base of every later run.

From the chapter, the beginning

Training starts with a perfect space and ruins it. Four instruments to see how — and why none of them alone is enough.

In the previous course, we tried semantic search with RoBERTa's embeddings. It failed in a specific way: every similarity came out above +0.99. Sentences about completely different subjects looked identical.

At the time we diagnosed it as anisotropy and moved on. Here the question became a different one: is this a defect of RoBERTa, or does it happen with any model? We measured it on ours.