People who believe "the model is the product" are potentially falling prey to a smuggled infinity.

"Simply make the model smarter and give it more tools and it can do everything."

A maximalist take.

"it will make better decisions than humans at some point, so even if it's tricked, it's OK because it's better than what a human would have been."

But that ignores the trolley problem kind of defaults dynamic.

A system that makes the same dangerous mistake a human likely would have, on behalf of a human who wasn't in the loop, is worse than the same mistake made by the human.