When you give a goal to a swarm, it creates a monkey paw situation due to Goodhart's law.

· Bits and Bobs 6/2/25
  • When you give a goal to a swarm, it creates a monkey paw situation due to Goodhart's law.
    • The goal is a metric and a metric is the map not the territory.
    • If you did it with aligned agents they'd do your intent not the letter where they disagree.
    • But if it's a swarm of unaligned agents with you, if the letter and the intent disagree they will go with the letter if it's more convenient for them.
    • Swarms of agents black boxing goals like "optimize my ad spend" will lead to bizarre grotesque results.

More on this topic

From other episodes