Not every website AB test is trying to win...
Sometimes in CRO / Experimentation, we test purely to not lose.
You would maybe be surprised how often we run experiments where the goal isn’t uplift.
It’s confirmation that a new element doesn’t tank conversions.
We might be:
- Rolling out accessibility improvements
- Rebranding a key page
- Simplifying layout to boost page speed
In these cases, we’re not chasing a shiny 12% lift . We just want statistical confidence that the change won’t hurt performance.
This kind of experiment is called a non-inferiority test.
Not sexy. Not headline-grabbing. But bloody important.
Because it’s often the tiny drops in conversion rate (the 0.3% here, the 0.7% there) that go unnoticed.
Until one day your website’s bleeding out from a thousand little cuts and the fingers start pointing (or worse they attribute the economy or a competitor).
That’s why we say: "any change made to a website should be tested" unless it's an obvious fix.
Sometimes, holding the line is the win.
Next step