An optimizing process will take an infinitesimal gain in the metric it's optimizing, even at a catastrophic loss on the metrics it's not paying attention to.
- An optimizing process will take an infinitesimal gain in the metric it's optimizing, even at a catastrophic loss on the metrics it's not paying attention to.[m]
- This is Stuart Russell's observation in the context of AI.
- Companies are optimizing machines to maximize shareholder value.
- The reason they don't run amok is that they're in competition with other maximizing machines: competitors.
- But think of these machines as being for the purposes of maximizing the net value created for society.
- Should a machine be allowed to create infinite value for its shareholders at the infinite cost to everyone else?
- Obviously not.