Russell's Catastrophe: when optimizing agents stripmine their blindspots.

Stuart Russell has pointed out that optimizing agents will trade a miniscule benefit in the dimension they're measured on, even if it causes a catastrophic loss in the dimensions they aren't measured on.