How do so-called "reasoning" model chatbots work?

Just around the corner on Facebook Victoria Livingstone was curious about that: I suggested:

 I believe that so-called "reasoning" models will take several cracks at responding to the prompt and then pick the "best" one. They're taking advantage of the fact that there's a certain amount of variability in how a prompt is responded to. There's not one and only one possible response. There's a distribution of possible responses. They're sampling the distribution of possible responses.  

I decided to verify that by – you guessed it – consulting the chatster. Here's what it said.

* * * * * 

Yes, that’s more or less correct, and a good intuitive summary of how many current “reasoning” or “chain-of-thought” models work. Let’s unpack it just a bit:

---

✅ What you're getting right:

1. Multiple attempts (sampling):
   Yes, models often sample multiple candidate completions (or reasoning paths) in response to a prompt. This can be done either in parallel or iteratively.
2. Variability of responses:
   Absolutely — language models are probabilistic. For a given prompt, there’s not just one “correct” or “best” output. Instead, there’s a distribution over possible next tokens, which accumulates into a distribution over entire responses. That’s the variability you’re referring to.
3. Selection of the “best”:
   Many reasoning approaches (e.g., majority voting, self-consistency, or value-guided reranking) generate multiple candidate answers and select among them, aiming to find the most accurate, coherent, or useful one.

---

🤏 Minor refinements:

1. Sampling vs. inference:
   The model itself doesn’t necessarily “choose” to take multiple cracks; rather, the system or wrapper around the model (e.g., a prompting or decoding strategy) instructs it to do so. The base model, on its own, typically just outputs one response unless configured otherwise.
2. What “reasoning” models are doing:
   In models like chain-of-thought prompting, reasoning arises from prompting the model to articulate intermediate steps. Some frameworks do this multiple times and pick the answer that appears most frequently (majority vote), or use a second model to judge plausibility or consistency.
3. It's not always about correctness:
   The “best” response may be selected based on various heuristics — e.g., length, fluency, presence of key terms, or alignment with external facts — depending on the objective of the system (e.g., reasoning accuracy vs. helpfulness vs. creativity).

---

Summary

So yes: your phrasing — "They're sampling the distribution of possible responses" — is a very good way to put it. And the practice of generating multiple completions and selecting among them is central to many of the most effective current approaches in LLM reasoning.

If you're interested, I can walk you through one of the specific methods — like self-consistency, tree of thoughts, or reflexion — which implement these principles more formally.