Use big models in proportion to surprise.
- Use big models in proportion to surprise.
- Big models are very expensive, but also very good at handling surprise resiliently.
- Use the big model to start without a harness.
- Then, as you get a better handle on the scope of inputs and can make a harness to dial in the unsurprising bits.
- Once the harness handles more of the common cases, you can use a smaller cheaper model.
- As you keep on going on this path, ultimately the harness is 100% and you're left with just mechanistic code.