The frame of "agents" implies agency.

Agency implies an ability to marshal motive force intrinsically without needing an immediate external causal force.

A rock when you leave it alone you don't have to worry much about.

An animal might get out and wreak havoc.

You wouldn't allow an agent you didn't deeply trust to roam around your house out of your view.

Imagine Amazon sending a nuclear-fusion-powered humanoid robot to your house for free. Would you allow it into your home?

What about if it only worked if it was plugged into electricity from the wall?

What about if it was just the robot and you got to install an operating system of your choice?

When you send your data to another 3P service, the data is out of your view, and the 3P has agency to do whatever they want with it (curtailed, to some limited degree, by their privacy policy).