LLMs are so noisy that it's hard to figure out the quality of underlying components built on top, because the noise dominates the signal.

· Bits and Bobs 5/5/25
  • LLMs are so noisy that it's hard to figure out the quality of underlying components built on top, because the noise dominates the signal.
    • Did it break because the surrounding system broke or because the LLM's behavior changed with an additional period that you added to the prompt?