Long-horizon reinforcement learning is hard.
In general, the shorter the feedback loop, the faster and easier the learning.
With long-horizon, you don't know until way later if the thing worked, and once you do you have to attribute the success across thousands of tokens.
Reward models can look at tokens in a long RL loop and decide if they matter or not.
Kind of like dopamine for models.
The reward models do a surprisingly good job, even though they're part of the same trained system.
They naturally pay attention to out-of-distribution or unambiguously important signal
Only recently did recipes for it that worked for long-horizon RL become known outside of major labs.
The knowledge about what ML recipes work diffuses over time.
The more publishing there is, the faster it happens.
But it will happen over sufficient timescales, as people switch companies and gossip.