Pond scum is emergently intelligent, but it can't speak to us.

But LLMs can, which is confusing!

We think of it as a thing, with a complex inner world, because it can speak to us and sound human-like.

This confusion leads to a category error.

It feels like a human made out of machines.

In reality it's more like a voice on top of a pond scum style emergent intelligence.

An intelligence that is absurdly good at absorbing mind-numbingly complex patterns.