Someone told me they played a game of 20 questions with an LLM, where the LLM thought of the secret item and the human asked questions.

It looks like the LLM picks an object and later we uncover more information on what it must be based on its answers.

But that's an illusion!

Actually the LLM has no idea what it "picked"; it's late binding its answers based on what seems most coherent with the things it's already said.

You and the LLM are discovering what the secret object was at the same time.

We project our normal human ways of thinking onto it because it sounds like a human.

But the way it works is in some ways very inhuman!