Why LLMs are good at generating code

13. Why do LLMs appear much better at generating code than generating general text?
Because, unlike the real world, the universe that a program manipulates (the state of the variables) is limited, discrete, deterministic, and fully observable.
The real world is none of that.

— Yann LeCun (@ylecun) February 13, 2023

After 50 years developing software I can definitely state that this is only true for the simplest cases. LLMs only appear to generate good code because they are only given the simplest cases and users are very forgiving of their mistakes.

— Marty Mouse (@MartyGargoyle) February 13, 2023

By comparison, see my old working paper, PowerPoint Assistant: Augmenting End-User Software through Natural Language Interaction (2013). Here's the abstract:

This document sketches a natural language interface for end user software, such as PowerPoint. Such programs are basically worlds that exist entirely within a computer. Thus the interface is dealing with a world constructed with a finite number of primitive elements. You hand-code a basic language capability into the system, then give it the ability to ‘learn’ from its interactions with the user, and you have your basic PPA.