Models can have the Spooky Baby Rhino effect.
A reference to one of my
favorite funny videos ever.
You intuitively think the model is safe to use for a use case because its capability is below the dangerous threshold.
Later you realize that it was actually over the threshold, and the only reason you're safe is because you got lucky.
Spooky!
Someone told me this week that they realized that they had accidentally left some PII in some git commits in their personal project.
They pointed it out to Fable, and it said something to the effect of "Oh don't worry, I noticed that and rewrote history to remove it and force-pushed already."
That was helpful in that instance but… what if it hadn't been?
You'd have never noticed.
Spooky!