Swarms give you monkey's paw dynamics.
- Swarms give you monkey's paw dynamics.
- When the incentives are divorced from the values, you get Goodhart's law.
- Swarms turbocharge this and give you efficiency on the incentives, at the cost of values.
- The swarm is an emergent force that is more powerful and impossible to control than any individual.
- This happens when the individuals optimize for the individual, not the collective.
- This is the default state, and must always be at least a little true.
- Swarms insulate people from the consequences of their actions.
- They make them more willing to do the action that is good for them directly but collectively add up to an obviously bad outcome.
- Swarms can be marshalled to do obviously terrible things.
- For example, imagine a betting market that invested tons of money in shorting "X will person will not die in the next few days."
- A powerful incentive for stochastic but targeted violence.
- Obviously that's an extreme example, but smaller examples of the same dynamic exist all over the place.