What is norm collapse in embeddings and how to detect it?

Norm collapse in embeddings is a situation where vector representations (embeddings) lose diversity in their length (norm) and converge to very similar norm values. This usually manifests as all embeddings having approximately the same length, which reduces their distinguishability and degrades the quality of search or classification.

Detecting norm collapse can be done by analyzing the distribution of norms (lengths) of embeddings. If the norm of most vectors is highly concentrated around a single value with low dispersion, it is a sign of norm collapse.

An example check in Python using numpy:

import numpy as np

embeddings = np.array([...])  # matrix of embeddings
norms = np.linalg.norm(embeddings, axis=1)
print(f"Average norm: {np.mean(norms)}")
print(f"Standard deviation of norms: {np.std(norms)}")

# If std is very small, it might indicate norm collapse