15 chunks · 11 episodes
Every swarm has an emergent Goodhart's Law. The individual player doesn't care about the collective so they take the edge to get a benefit. This happens to the extent the players don't have a strong shared belief in the collective. AI runs swarms even harder, which will lead to even more Goodhart's
The Principal Agent Problem is a specific subclass of the broader Goodhart's Law phenomenon. The difference between the actor's incentives and the incentive of the collective leads to an emergent outcome. There must be some difference between the incentives of the two… and possibly quite a lot of di
There's a class of shortcuts that are good for the individual but bad for the collective. Those are what will be taken, according to Goodhart's Law. That's the gradient that the swarm will follow. An individual who doesn't care about the collective (or feels that it doesn't make sense for them to do
James Evans gave a fascinating talk I attended this week. He studies how collectives "think." Unpredictability is the best predictor of a paper being highly influential.[hp] These ideas are "off manifold," they are outside the normal landscape of research. As AI makes it possible to work with large
Optimizers will choose gains on the target metric at catastrophic cost to unmeasured externalities. Another way of describing the fundamental reason Goodhart's Law shows up.
Excellent piece from Ben Mathes on Goodhart's Law and "Lowest Common Consensus". Why organizations tend to focus on a simple, obvious metric, and then over-focus on it. It's simply easier to agree what metric to use if everyone agrees it's important.
Hyper financialism is just Goodhart's Law. In that mindset there is nothing other than "make number go up". All humanity, all taste, all meaning has been hollowed out. The shortcut is the point, there is nothing else. We made capitalism and politics so "efficient" that we Goodhart's-lawed ourselves
Dimensions omitted from an optimization target will be set to the worst possible value. This is a provable outcome. Optimizing focuses on some dimensions to the exclusion of everything external. If setting to the worst value in the untracked dimension creates even an infinitesimal edge in the tracke
It's not that Goodhart's Law just so happens to find shortcuts. It's that the gradient the swarm descends fundamentally are shortcuts. The ideal vector is what all of the members of the collective, if they didn't know which member of the swarm they were, would pick. The veil of ignorance, where the
Swarms are adaptable but have Goodhart's Law. The antidote is trust in the collective and long-term goals. When individuals trust each other to behave as a collective they believe in, they will take actions that don't follow Goodhart's Law and don't destroy the collective. Instead of only optimizing
I like the frame of "workslop". Performative use of AI at work creates more work for others as an externality.[g] It's easier to generate workshop than to respond to it. If there's a top-down mandate to use AI, then most of the use will slow your organization down. Goodhart's Law strikes again.
For evals to give you a gradient of improvement, the eval has to not be saturated. If it's saturated then the gradient gives you Goodhart's Law. It pulls you towards optimizing something that does not actually improve what you care about.
Cory Doctorow shared the story of a startup that is "Uber for Nursing." The service buys the credit reports of nurses on the platform, and offers nurses who have debt a lower fee for completing jobs. This is gross–it's not about "how able is this person to pay back a debt to me" but about "what is t
LLMs are great at debunking… but also bunking. So if it has intimate knowledge of you and is not perfectly aligned (an impossibility) you get Goodhart's Law. An epic, society-scale monkey's paw. Hold on to your butts!
One way to mitigate Goodhart's Law: keep the actual objective secret[mt]. Then, swap in an ever-changing set of proxy metrics. You could argue that good CEOs do this–explicitly or implicitly.[mu]