When you give a goal to a swarm, it creates a monkey paw situation due to Goodhart's law.
- When you give a goal to a swarm, it creates a monkey paw situation due to Goodhart's law.
- The goal is a metric and a metric is the map not the territory.
- If you did it with aligned agents they'd do your intent not the letter where they disagree.
- But if it's a swarm of unaligned agents with you, if the letter and the intent disagree they will go with the letter if it's more convenient for them.
- Swarms of agents black boxing goals like "optimize my ad spend" will lead to bizarre grotesque results.