Optimizing… but for what?

Due to Goodhart's law it's not what you think it is.

We treat optimization like an unalloyed good.

In the right situations it can be hugely beneficial.

But as it saturates it starts to go wonky and decohere from what we actually care about.

Just some number that lots of people agree is worthwhile, because everyone else agrees it's worthwhile.

Creating hollow improvement at the cost of resonant quality.