Alison Gopnik has a wonderful metaphor: stone soup AI.

In the stone soup story, a traveler seeks to make a soup out of only a stone.

He says that a small garnish will make it taste much better, and their neighbor obliges.

The garnish is minor compared to the existing quality of the soup so it makes sense to invest it.

He keeps stepping up the size of incremental ingredients to add, which incremental neighbors are happy to do because the underlying soup has ratched up its quality so investing in it incrementally more seems plausible.

The result is a delicious soup that everyone marvels is "just made from a stone"

Their own incremental additions all felt less important than the pre-existing soup when they were added to it, so they seem unimportant.

But the totality of the "garnishes" is large.

The soup gets its taste from all of the ingredients, not the stone.

AI isn't too dissimilar.

The transformer architecture is a "simple" web of simulated neurons: a stone.

But then if you add to that stone soup all of culture and society, out pops an amazingly tasty dish.

But the reason the dish is so tasty is not the stone, it's all of the other ingredients!