Next time someone tell you that subscription AIs don't hallucinate at all anymore or have greatly reduced the problem - ask them to Google it. Or give them these stats with citations : According to the First Page Sage 2026 Report, on subscription models Claude has a response-level hallucination rate of 8.3% and ChatGPT sits at 11.0%, Volatile real-world details. Questions about local business hours and addresses trigger hallucinations 26.0% of the time, while breaking current events sit at 21.0% and specific product pricing sits at 19.4%. Across major platforms, queries about niche subjects produce a 31.0%hallucination rate, making them 4.4 times more error-prone than simple factual lookups at 7.0%. Questions containing a false premise are the worst performers, triggering fabrications 46.1% of the time as models play along with the user's mistake. Obscure public figures and trivia. OpenAI's own system cards show that answering short factual trivia from memory causes hallucinations 40% to 51% of the time on flagship reasoning models, while asking about lesser-known public figures triggers errors 33% of the time. High-stakes domains. General legal questions produce fabricated statutes or case law 23.5% of the time, though specialized audits of federal court cases have recorded rates as high as 58%. Financial queries average a 16.8% error rate, medical and health queries sit at 15.0%, and software coding queries average 12.1%. When tested on 6,000 difficult factual questions without internet access on the Artificial Analysis AA-Omniscience Leaderboard, hallucination rates rise dramatically because models guess when they miss. OpenAI's GPT-6.1 Terra hallucinates on 34.5% of its non-correct answers, GPT-6 Astra hits 49.8%, and GPT-6.1 Sol reaches 51.9%. Anthropic's Claude Sonnet 5.5 hallucinates on 43.3% of misses, Claude Opus 5.5 hits 50.3%, and Claude Fable 5.1 reaches 70.0%despite achieving the highest raw accuracy. https://t.co/gFCEgjiqI7
— Ewan Morrison (@MrEwanMorrison) November 18, 2026
Wednesday, November 18, 2026
AI Hallucination Rates
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment