The most expensive misunderstanding in applied AI is the quiet assumption that a model which sounds certain is certain. Fluency is not knowledge. A large language model produces the most probable continuation of a prompt; it has no internal signal that separates "this is well-supported" from "this is plausible and entirely invented." The confidence is in the prose, not in the facts.
This is not a bug awaiting a patch in the next version. It is what the system is. A model generalizes from what it has seen and degrades — silently, fluently — the moment the input drifts outside that distribution. It will not tell you it has left familiar ground. It answers with exactly the composure it had when it was right. The failure mode of a probabilistic system is not an error message. It is a confident wrong answer that looks identical to a confident right one.
The practical consequence: you cannot make a model reliable by trusting it more carefully. Reliability has to come from around it — retrieval that grounds it in something checkable, explicit boundaries on what it is permitted to decide, validation that catches the confident-wrong case before it reaches a customer. The model is an input to a system, not the system.
Treating "the model is impressive" as "the model is dependable" is how demos become incidents. The demo lives inside the distribution the model knows. Production is the part of the world the model has never seen — and that is exactly where the user will take it.