A/B Testing: How to Get It Right

Tags

Top Highlights

• determining which short-term metrics are the best predictors of long-term outcomes is difficult.

• It requires close cooperation between senior executives who understand the strategy and data analysts who understand metrics and trade-offs. And it’s not a onetime exercise: We recommend that the OEC be adjusted annually.

• Getting numbers is easy; getting numbers you can trust is hard!

• You need to allocate time and resources to validating the experimentation system and setting up automated checks and safeguards.

• In certain cases a single good or bad segment can skew the average enough to invalidate the overall results.

• assess its impact

• assess the potential of new ideas

• benefit of having a capability for running many tests cheaply and concurrently

• experiment with everything

• At a time when the web is vital to almost all businesses, rigorous online experiments should be standard operating procedure

• assess not only ideas for websites but also potential business models, strategies, products, services, and marketing campaigns—all relatively inexpensively

• Controlled experiments can transform decision making into a scientific, evidence-driven process—rather than an intuitive reaction

• breakthroughs might never happen, and many bad ideas would be implemented, only to fail, wasting resources

• haphazard in their experimentation approach, don’t know how to run rigorous scientific tests, or conduct way too few of them

• A/B

• Any company that has at least a few thousand daily active users can conduct these tests

• evaluate many ideas quickly, with great precision, and at a negligible cost per incremental experiment

• iterate rapidly, fail fast, and pivot

• Tiny changes can have a big impact.

• success is more about getting many small changes right