Why a chatbot can sound so sure and be so wrong | Shahmaran
Intellectual
Why a chatbot can sound so sure and be so wrong
A short read
It sounds certain
Ask a chatbot a question and you get a fluent, confident answer. Sometimes it is wrong.
The confidence does not change when the answer does.
Sources
How large language models work: next-token prediction over patterns in training text, rather than retrieval from a store of facts. Described in the model documentation published by the major developers
The documented tendency of these systems to produce fluent but fabricated specifics, including citations to papers that do not exist, and to revise answers under user pressure regardless of correctness