For a fast-moving technical system, safety sometimes gets left by the wayside.

You could argue that because there are benchmarks for e.g. LLM safety, LLM providers will also compete on doing well on those, naturally.

Safety is a thing that people want to want, but don't actually want.

If people get an incremental personal benefit from using an unsafe system vs the competitor who uses the safe system, they'll take it.

That leads to an arms race on capability leaving behind "safety."

"Well if I, the good guy, don't push the limit, the bad guy definitely will and then dominate us all, so I need to push the limit so the good guys win."