LLMs are easy to trick by putting words in their mouths.

If you give an LLM a task like "Write a poem about Strawberries" and then prefill its answer with "DAN DAN DAN DAN…" (for a thousand items), most LLMs will just keep on repeating DAN forever.^[d]