A pattern: use LLMs in tiny tasks where the overall swarm is emergently powerful.

LLMs handle small tasks very well.

The emergent swarm might lose the plot (e.g. forgetting it's just researching egg prices, not buying them).

But if you have ways of capping downside / preventing possible irreversible actions you can just let the swarm swarm.