Last week I talked about the benefits of A/B testing infrastructure.

Afterwards I had a few conversations with folks about some of the downsides of an A/B testing infrastructure.

One person described a downside as "p-hacking on steroids"

Another downside is that it can create worse externalities.

A/B testing is a great way for hill-climbing a metric. A/B testing accelerates Goodhart's law.

If there are externalities that are not incorporated in your measurement loop (which there fundamentally must be) you might find that climbing the A/B test hill is actually creating small amounts of value on the hill but significantly more negative externalities.

Worse, these externalities, being outside of your measurement loop, will lurk and be invisible to you until they fully consume you.