When was the last time you got accidentally tricked by a hallucinated fact from an LLM and didn't catch it?

You look at its response and think "Look, it knew the answer to X" then as you look more closely you realize it just hallucinated something plausible and you didn't even notice.

Even when you know it can happen, you don't think to check.

Because by definition its hallucinated answers all look totally plausible, so they don't flag your quick smoke test glancing at them.