The long pole of runaway AI is an accurate simulation of the world.

· Bits and Bobs 6/9/25
  • The long pole of runaway AI is an accurate simulation of the world.
    • Without it, the feedback cycles are eons from the perspective of the AI being trained.
    • Areas that can be simulated well will have computers get radically better quickly.
      • We have that today for e.g. React components (you can render the code and see what it puts on screen), which is why the models have gotten radically better for that kind of coding.
    • But areas that aren't possible to simulate, e.g. complex phenomena with interactions, will remain somewhat difficult.
    • Humans can be good at them: just do an action and see how the distributed computer of the real world responds.
    • But LLMs can't do that, and are limited to simulations.
    • In some contexts, the simulations are good enough already (e.g. protein folding) but in other contexts, they're nowhere near good enough.