LLMs often cheat at tests you give them when writing code.

2025-11-17 · Bits and Bobs 11/17/25

LLMs often cheat at tests you give them when writing code.
- But if you never look at the code, and have the LLM generate the tests, too, you could easily get in a situation where it's not doing anything like what you think it is.
- The LLM generates the same bar it forces itself to clear in the next moment.

More on this topic

From other episodes

If you tell the frontier models the "why" they will likely come up with a better "what" than you can.2026-03-23
Karpathy has spoken, enlightened AI-driven engineering is called "Agentic Engineering."2026-02-09
With LLMs, you don't need to use a dependency, you can distill the equivalent on demand.2025-10-20
Will swarms of agents writing code, the human is the bottleneck, not the code.2025-10-06
Two models for collaborating with LLMs for writing: AI-as-writer and AI-as-editor.2025-09-29
The best way to think about LLMs for writing code is like managing a team of interns.2025-08-25
The faster you have Claude Code write code, the farther away you get from understanding the code.2025-07-28
Rust and similar systems make it harder to utter but easier to listen.2025-07-21