27 chunks · 17 episodes
Goodhart's Law emerges when the players don't care about the game as an end in and of itself.[n] When not aligned with the end, the smarter the players, the more quickly they destroy value.
A Twitter post: Don't trust your agents. On Autoresearch and overfitting. If you just have the agents optimize, they'll overfit to the problem domain. Goodhart's law shows up even without humans.
Every swarm has an emergent Goodhart's Law. The individual player doesn't care about the collective so they take the edge to get a benefit. This happens to the extent the players don't have a strong shared belief in the collective. AI runs swarms even harder, which will lead to even more Goodhart's
The Principal Agent Problem is a specific subclass of the broader Goodhart's Law phenomenon. The difference between the actor's incentives and the incentive of the collective leads to an emergent outcome. There must be some difference between the incentives of the two… and possibly quite a lot of di
Optimizing… but for what? Due to Goodhart's law it's not what you think it is. We treat optimization like an unalloyed good. In the right situations it can be hugely beneficial. But as it saturates it starts to go wonky and decohere from what we actually care about. Just some number that lots of peo
There's a class of shortcuts that are good for the individual but bad for the collective. Those are what will be taken, according to Goodhart's Law. That's the gradient that the swarm will follow. An individual who doesn't care about the collective (or feels that it doesn't make sense for them to do
James Evans gave a fascinating talk I attended this week. He studies how collectives "think." Unpredictability is the best predictor of a paper being highly influential.[hp] These ideas are "off manifold," they are outside the normal landscape of research. As AI makes it possible to work with large
Optimizers will choose gains on the target metric at catastrophic cost to unmeasured externalities. Another way of describing the fundamental reason Goodhart's Law shows up.
Excellent piece from Ben Mathes on Goodhart's Law and "Lowest Common Consensus". Why organizations tend to focus on a simple, obvious metric, and then over-focus on it. It's simply easier to agree what metric to use if everyone agrees it's important.
Hyper financialism is just Goodhart's Law. In that mindset there is nothing other than "make number go up". All humanity, all taste, all meaning has been hollowed out. The shortcut is the point, there is nothing else. We made capitalism and politics so "efficient" that we Goodhart's-lawed ourselves
Dimensions omitted from an optimization target will be set to the worst possible value. This is a provable outcome. Optimizing focuses on some dimensions to the exclusion of everything external. If setting to the worst value in the untracked dimension creates even an infinitesimal edge in the tracke
It's not that Goodhart's Law just so happens to find shortcuts. It's that the gradient the swarm descends fundamentally are shortcuts. The ideal vector is what all of the members of the collective, if they didn't know which member of the swarm they were, would pick. The veil of ignorance, where the
Swarms are adaptable but have Goodhart's Law. The antidote is trust in the collective and long-term goals. When individuals trust each other to behave as a collective they believe in, they will take actions that don't follow Goodhart's Law and don't destroy the collective. Instead of only optimizing
I like the frame of "workslop". Performative use of AI at work creates more work for others as an externality.[g] It's easier to generate workshop than to respond to it. If there's a top-down mandate to use AI, then most of the use will slow your organization down. Goodhart's Law strikes again.
For evals to give you a gradient of improvement, the eval has to not be saturated. If it's saturated then the gradient gives you Goodhart's Law. It pulls you towards optimizing something that does not actually improve what you care about.
Goodhart's law is a form of reward hacking. The swarm doesn't care about the end, only the metric.
When you demo a product, you can Goodhart's law yourself. When you use a product, you stick to the paths that will be most useful. To demo a thing you intuitively stick to the paths that work, not that are useful. You Goodhart's law yourself to think it's working by focusing on what demos well. You'
The modern industry selects for quickness of reasoning, not depth. The fastest one-ply thinkers. That's what gives us Goodhart's law at the level of society; everyone thinking a single ply and not thinking at all about the implications of their actions. One-ply thinkers who are handsomely rewarded m
Harvard Business Review: What Gets Measured AI Will Automate. Goodhart's law on steroids. Paperclip maxing ourselves to death. Heaven help us.
Cory Doctorow shared the story of a startup that is "Uber for Nursing." The service buys the credit reports of nurses on the platform, and offers nurses who have debt a lower fee for completing jobs. This is gross–it's not about "how able is this person to pay back a debt to me" but about "what is t