A single metric can never capture the real world nuance.

· Bits and Bobs 8/25/25
  • A single metric can never capture the real world nuance.
    • Data scientists and finance people typically look at a portfolio of metrics that are all correlated in different partial ways with what they actually care about, and then triangulate.
    • In that regime, there must be someone to ultimately make a judgment call about which metrics matter at a given time.
    • That person making the call can draw on their nuanced experience and context, and should have long-term alignment and responsibility for the outcome.
    • The tech industry seems relatively naive in this domain.
    • The tech industry tends toward "figure out a single objective metric and simply make it go up."
      • There have been formative experiences like when the newsfeed switched to algorithmic, where users protested loudly, but the metrics showed that users loved it.
      • This led to a mindset in the industry of "the users don't know what they want so just laser focus on the metric."
      • But that's the wrong lesson!
      • The metrics can only show what the limbic system wants, not what the user wants to want.
    • The search for a single "objective" metric precludes capturing complexity and nuance.
    • Each individual metric is a half-truth.
    • Which metric to pay attention to in a given moment is a judgment call… you can either be explicit about that or ignore that.

More on this topic

From other episodes