More important than model distillation is code distillation.

Use LLMs to distill mechanistic software in proportion to the amount of surprise.

LLMs are great at handling surprise, but if the same input comes in a lot, that should be dispatched to mechanistic code with the LLM branch as self-healing fallback.

Some architectures make this much easier: for example an open-ended set of labels, storing positive and negative examples from the real query stream so mechanistic matching has a test set, the ability to share distilled lists of matching statements safely and anonymously with others, etc.

The right meta-structure enables auto-distilling software that automatically gets more efficient as it's used.