I saw another demo of Github's Copilot workspaces.

It allows an agent, with some continuous babysitting from a human, to plan a change, research it, then modify the necessary files and generate a pull request.

One of the reasons it works is because it's not fully autopilot; the human can nudge and redirect it throughout the process, including adding files it should include in its plan but didn't.

Smoothly marbled human-in-the-loop throughout the flow.

If it gets a little off track, you can correct it before it gets too far down the wrong path when it's harder to correct.

Imagine a few more steps: Agents filing github issues. Agents watching other agents' Copilot workspaces to give each other feedback on how to improve. GitHub Actions continuously deploying.

Agents fixing and improving programs that affect the real world.

An auto-hoisting loop that affects the real world.

And then… oops! Singularity!