Harnessing the Universal Geometry of Embeddings

Rishi Jha, Collin Zhang, Vitaly Shmatikov, John X. Morris, Harnessing the Universal Geometry of Embeddings, arXiv:2505.12540v2 [cs.LG] https://doi.org/10.48550/arXiv.2505.12540

Abstract: We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets.

The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

 The conclusion:

8 Discussion and Future Work

The Platonic Representation Hypothesis conjectures that the representation spaces of modern neural networks are converging. We assert the Strong Platonic Representation Hypothesis: the latent universal representation can be learned and harnessed to translate between representation spaces without any encoders or paired data.

In Section 5, we demonstrated that our vec2vec method successfully translates embeddings generated from unseen documents by unseen encoders, and the translator is robust to (sometimes very) out- of-distribution inputs. This suggests that vec2vec learns domain-agnostic translations based on the universal geometric relationships which encode the same semantics in multiple embedding spaces.

In Section 6, we showed that vec2vec translations preserve sufficient input semantics to enable attribute inference. We extracted sensitive disease information from patient records and partial content from corporate emails, with access only to document embeddings and no access to the encoder that produced them. Better translation methods will enable higher-fidelity extraction, confirming once again that embeddings reveal (almost) as much as their inputs.

Our findings provide compelling evidence for the Strong Platonic Representation Hypothesis for text- based models. Our preliminary results on CLIP suggest that the universal geometry can be harnessed in other modalities, too. The results in this paper are but a lower bound on inter-representation translation. Better and more stable learning algorithms, architectures, and other methodological improvements will support scaling to more data, more model families, and more modalities.

For a variety of reasons I am strongly sympathetic to this Strong Platonic Representation Hypothesis. It seems to be the sort of thing one would predict on the basis of the arguments I have made about the relationship between the latent structure of an LLM and what I am calling the metaphysical structure of the universe: See pp. 34-38 and 42-44 in ChatGPT: Exploring the Digital Wilderness, Findings and Prospects (2025).