Output-based pricing will likely have a Goodhart's law problem.
- Output-based pricing will likely have a Goodhart's law problem.
- The provider will want to just get done precisely what the customer asked for.
- Similar to the monkey's paw.
- It's impossible to specify precisely what you want against a malicious implementer.
- The time to specify details compounds super-linearly.
- That means that many problems quickly go underwater where it's cheaper to simply do it yourself than to specify it for others to execute.
- You don't need a malicious implementor, just an optimizing one.
- They'll take even a marginal benefit on the dimension they're being graded on at a catastrophic cost on a dimension they aren't being grade on.