The eval loop is the beating heart of a quality improvement process.

· Bits and Bobs 4/14/25
  • The eval loop is the beating heart of a quality improvement process.
    • For example, a search quality problem.
    • The loop: sample sessions that had a bad result, come up with scalable ways of improving them, experiment, ship, repeat.
    • At the very beginning it's hard to get that eval loop humming.
      • You have to make it turn a few times before it gets going under its own steam.
      • Kind of like hand-crank starting a car motor.
    • The eval loop can absorb all of the attention and resources you give it.
      • It can absorb infinite energy.
    • So be careful to only give it the proper amount of attention.
    • If the core product has PMF and you're still getting super-linear returns from the loop, then keep investing more in it.
    • But if the eval loop is for a secondary part of the product, or a product that doesn't yet have PMF, or is getting significantly diminishing returns because you're hitting the quality asymptote, pull back resources.

More on this topic

From other episodes