I love this talk from Steve Strassmann: Why AIs get Cancer.
- I love this talk from Steve Strassmann: Why AIs get Cancer.
- Cancer is not merely a metaphor for AI dysfunction.
- It's the same underlying governance disease.
- Any system with
- 1) a group/individual hierarchy
- 2) evolutionary pressure on the individual, and
- 3) the ability for individuals to modify group rules
- will eventually produce cancer-like pathology.
- Evolution is "the ultimate jailbreak algorithm."
- It punches through any external barrier.
- We use it everywhere: sports, markets, elections, AI benchmarks.
- But Goodhart's Law corrupts every contest: the metric becomes the target and ceases to measure what it was meant to.
- When cells formed multicellular organisms ~1B years ago, external survival pressure shifted to the group while individuals got a new pressure: compliance with group rules.
- This creates two competing evolutionary cycles in tension.
- As long as they stay in balance, the system works.
- When individuals evolve to break the rules you get "cancer".
- And of course, evolution selects for rule-breaking.
- Intelligence/Acceleration = going faster, optimizing the metric, winning the contest.
- Wisdom/Governance = steering, brakes, asking "where are we driving and for whom?", protecting the group from the individuals trying to beat the system.
- The root cause of cancer is short-sightedness[o].
- Optimizing observable short-term metrics while ignoring long-term systemic consequences.
- What counts as disease is different at the individual and collective level.
- Algae blooms are disease states for the ecosystem, but not for the individual algae organisms.