It's easy to trick yourself when optimizing for metrics.

A real world example, a second hand story from Google Maps.

When viewing a place page, there's a button to get directions, with a big obvious icon.

The text beneath the icon used to be how long it would take to drive there, if you left right then: "15 minutes"

When experimenting they found, somewhat surprisingly, users seemed to like it better if the text instead said "Get Directions".

But this improvement was an illusion.

The "user satisfaction" proxy was CTR, and CTR did indeed go up when the text was changed to "Get Directions".

But that was not because users understood the button better.

Instead, it was because a class of users who previously had their needs met without clicking the button ("Do I need to leave now?") now had to click through to get their information.