determining which short-term metrics are the best predictors of long-term outcomes is difficult.
It requires close cooperation between senior executives who understand the strategy and data analysts who understand metrics and trade-offs. And it’s not a onetime exercise: We recommend that the OEC be adjusted annually.
Getting numbers is easy; getting numbers you can trust is hard!
You need to allocate time and resources to validating the experimentation system and setting up automated checks and safeguards.
In certain cases a single good or bad segment can skew the average enough to invalidate the overall results.
assess its impact
assess the potential of new ideas
benefit of having a capability for running many tests cheaply and concurrently
experiment with everything
At a time when the web is vital to almost all businesses, rigorous online experiments should be standard operating procedure
assess not only ideas for websites but also potential business models, strategies, products, services, and marketing campaigns—all relatively inexpensively
Controlled experiments can transform decision making into a scientific, evidence-driven process—rather than an intuitive reaction
breakthroughs might never happen, and many bad ideas would be implemented, only to fail, wasting resources
haphazard in their experimentation approach, don’t know how to run rigorous scientific tests, or conduct way too few of them
Any company that has at least a few thousand daily active users can conduct these tests
evaluate many ideas quickly, with great precision, and at a negligible cost per incremental experiment
iterate rapidly, fail fast, and pivot
Tiny changes can have a big impact.
success is more about getting many small changes right
Glasp is a social web highlighter that people can highlight and organize quotes and thoughts from the web, and access other like-minded people’s learning.