NEW SAVANNA: Once more around the block: LLMs are trapped by their training data, which defines the boundaries of THEIR world; they can't generalize beyond it (very well).

Monday, November 6, 2023

Once more around the block: LLMs are trapped by their training data, which defines the boundaries of THEIR world; they can't generalize beyond it (very well).

Part of the confusion is that in a space as rich as natural language, in-distribution, out-of-distribution, & generalization aren't well-defined terms. If we treat each query string as defining a separate task, then ofc LLMs can generalize. But that's not a useful definition.
— Arvind Narayanan (@random_walker) November 6, 2023

I suspect what happened here is that many people have been gradually revising their expectations downward based on a recognition of the limits of GPT-4 over the last 8 months, but this paper provided the impetus to publicly talk about it.
— Arvind Narayanan (@random_walker) November 6, 2023

Re. the "b-b-b-but this paper doesn't show…" replies: I literally started by saying this paper isn't about LLMs. My point is exactly that despite being not that relevant to LLM limits the paper seems to have gotten people talking about it, perhaps because they'd already updated.
— Arvind Narayanan (@random_walker) November 6, 2023

Here's the paper they're talking about: