Some kinds of systems are black boxes to you because you can't interact with the system directly, but only through some limited intermediary.

Someone could reach in and tweak it and control it inside the black box, just not you.

Any proprietary system has some aspect of its behavior that is a black box outside of the company.

But some systems are black boxes to all humans.

Even if you had access, you wouldn't be able to understand or control it, because its workings are fundamentally not grokable by humans.

They are black boxes to everyone due to their fundamental nature.

LLMs are these kinds of fundamental black boxes.