Even (very) noisy LLM evaluators are useful for improving AI agents.
- Another example of noisy-but-biased signal still giving a gradient if you have enough of it.
From other episodes
- End-to-end encryption in messaging is now table stakes.2026-04-06
- If a system is open but so different from everything else, it is effectively closed.2026-02-23
- The gradient of competition can only give users what they want, not what they want to want.2025-12-08
- If you follow the gradients of optimization you get what people "want" not what they "want to want."2025-11-24
- There's a class of shortcuts that are good for the individual but bad for the collective.2025-11-17
- Enshittification is inevitable for successful products.2025-11-10
- It's not that Goodhart's Law just so happens to find shortcuts.2025-10-27
- Data can only tell you the gradient, not where to go.2025-10-27