Goodhart's law emerges out of swarms.

· Bits and Bobs 5/5/25
  • Goodhart's law emerges out of swarms.
    • In a swarm, each individual in the swarm follows their local incentives, not the incentive that the collective cares about.
    • To the extent that an agent in the swarm cares about the collective as an intrinsic good, they can do what's right for the collective even if it's incrementally worse for them.
      • The more shame they feel about going against the collective goal, the more likely they are to optimize for the collective.
    • But if the individuals feel no allegiance, they'll simply follow whatever their personal incentive is.
    • The swarm behavior doesn't arise because of human nature, it arrives because of the independent actors with interdependent decisions.
    • LLM agents swarming will cause Goodhart's law even faster, because they will have even less alignment with the collective, and don't feel shame.
      • Fully transactional and consequentialist, and also don't fear being knocked out of the game.