Certain systems are monkey's paw systems.

If there is any way to interpret it in a way that is not aligned in your interest, it will.

A genie might naively misinterpret your request by filling in details in ways you didn't expect.

But a monkey's paw will maliciously misinterpret.

Goodhart's Law exists everywhere but is often checked by forces like the agents having long-term belief in the shared goal.

Monkey's Paw systems have no countervailing force; it's Goodhart's Law to the max.

LLMs are by default like a genie… but when they've been prompt injected they're a monkey's paw.