Harvard Business Review: What Gets Measured AI Will Automate.
Harvard Business Review: What Gets Measured AI Will Automate. Goodhart's law on steroids. Paperclip maxing ourselves to death. Heaven help us.
29 chunks · 19 episodes
Harvard Business Review: What Gets Measured AI Will Automate. Goodhart's law on steroids. Paperclip maxing ourselves to death. Heaven help us.
Cory Doctorow shared the story of a startup that is "Uber for Nursing." The service buys the credit reports of nurses on the platform, and offers nurses who have debt a lower fee for completing jobs. This is gross–it's not about "how able is this person to pay back a debt to me" but about "what is t
LLMs are great at debunking… but also bunking. So if it has intimate knowledge of you and is not perfectly aligned (an impossibility) you get Goodhart's Law. An epic, society-scale monkey's paw. Hold on to your butts!
Goodhart's law is a form of 'cheating'. Cheating happens with agents who aren't aligned with the collective as an end in and of itself. That means if there's an action that will get them as an individual an edge at the cost of the collective, they'll take it. You can get strong alignments by having
When you give a goal to a swarm, it creates a monkey paw situation due to Goodhart's law. The goal is a metric and a metric is the map not the territory. If you did it with aligned agents they'd do your intent not the letter where they disagree. But if it's a swarm of unaligned agents with you, if t
Goodhart's law is what leads to business models being misaligned with consumers in the limit. Companies are a swarm of employees making interdependent decisions whose ground truth is the business model. Ultimately the ground truth about what a company cares about is the business model. Everything el
One way to mitigate Goodhart's Law: keep the actual objective secret[mt]. Then, swap in an ever-changing set of proxy metrics. You could argue that good CEOs do this–explicitly or implicitly.[mu]
"Reward hacking" in models is just a specific example of Goodhart's law. If you get a result from a system that you can't understand (that is a black box to you), you can't check to see if it's found a deep real pattern, or something superficial. Apparently there was an example where a "tank detecto
An alternate frame of Goodhart's law: you can either steer the ship or understand the ship.[tf][tg] The uncertainty principle applied to organizations. I heard this formulation from Ade Oshineye.