Goodhart's law shows up in complexity.

Imagine Goodhart's law as the streetlight fallacy.

You optimize the value that is internal, captured by the model, in the light of the streetlight.

You ignore whatever is in the dark (the externalities) even if it is larger and more important.

Computers will optimize for the objective metric, mercilessly.

They will create the value under the streetlight, at the cost of destroying the value that's in the dark.

In complex environments, most of the value is in the dark.

So a hyper-optimizer will destroy value if you aren't careful.

The problem in this situation is not the computer, the problem is having a single entity that doesn't have to compete with others.

It is imperative that we don't centralize the power of AI algorithms but keep them federated and competing, keeping one another grounded and in check.