LLMs aren't great at counterfactuals, for the same reason they aren't great at writing.
- LLMs aren't great at counterfactuals, for the same reason they aren't great at writing.
- To do well at both requires stably inhabiting a very different perspective than your own current perspective.
- For writing, it's to figure out "what is my reader thinking and how do I help guide them to think the thing I want them to."
- For counterfactuals, it's "if we change this one thing, how would that play out in the world?"
- Humans forget easily and only have an approximate sense of our current situation, so it's easy for us to intentionally dissociate temporarily.
- But LLMs have a more broad, formal sense of situation and coherence with everything they were trained on.
- That keeps pulling them back to what they know about reality.
- Humans are more feeble; we do fuzzier thinking, so it's easier to cut ourselves loose.