The expression of nuance and high dimensionality in LLMs

Jonathan Falk quoted over at Statistical Modeling, Causal Inference, and Social Science in a post by Andrew:

I have spent 50 years fighting the Curse of Dimensionality. I know this curse in my marrow. Brilliant inferences await me, but the space in which these insights are found is simply too vast to explore. So we simplify, reducing the dimensionality to something that while still vast, is confined to a hyperplane where we can, like Plato, see the projections of truth, not the truth itself.

But then what LLMs and their generation have taught me is that nuance, which is really just the inverse of inference (in that it’s the vast set of all things consistent with some inference) has an amazing boon of dimensionality. There appears to be no thought that can’t be described by a 14,000 dimension or so vector whose tuning has the huge advantage that 14,000-dimensional space is so empty that tiny nuances can be readily distinguished in such a space, so that you can hide uniqueness in the vastness of 14000-dimensional space that you couldn’t recover in a raw search in that same space.