The Experimenter
Shortly after Thomas Edison went through ten thousand types of filament before landing on Tungsten as the best choice for his light bulb, a reporter asked him if he felt bad for “failing” ten thousand times. Edison simply responded that he had not failed at anything, but in fact had learned ten thousand things that did not work. The difference between failure and learning, he said, was that “Experience is failure with learning applied.”
When deciding between site changes—whether a design decision or a business process—decisions are often made by opinions or feelings, rather than data. This isn’t always a bad thing, especially when an inspired leader who knows their business can really connect with customers in the online world. The rest of us need a mechanism to gauge whether or not a new lever is moving us forward, and provide enough of a fact-base to say with certainty if something is working.
Prototyping and Controlled Experimentation are two of the best methods to see if competing ideas work better. Examples from my client base include the following:
When Dell wanted to increase the sale of extended warranties on their personal computers, we used an A/B experiment to see which default setting (1yr, 2yr, 3yr) on which model (low end, medium range, high end) would produce the highest overall margin. The test yielded the optimal margin warranty mix.
A “refer a friend” email campaign used a prototype naming convention for the email links to describe the type of offers and their locations within the body of the email.
eLuxury, an online luxury retailer specializing in high end women’s apparel desired to know if the high cost of using models to display handbags resulted in enough incremental sales and fewer returns than shooting the handbags without models, therefore offsetting the increased marketing cost.
Wells Fargo wanted to test the clickthrough rates of several offer placements for their redesigned home page. A multivariate test with different design treatments (rollover, flash, text) was used to determine which treatments made the home page offers more compelling.
A large financial services company used controlled experimentation to determine whether images on the bottom of the page had a higher clickthrough rate than the same images on the right hand side of the page. This straightforward A/B test resulted in a clear distinction between treatments and helped establish design standards for the site.
(Click the above image to open in larger window)
Controlled Experimentation Lifecycle
- Exploratory Data Analysis – Something has to point you in the direction of what to test, right? Whether data-driven or from an executive, ideas can come from anywhere.
- Create Hypothesis – The most important part of the experiment, defining what you want to measure and what your success metrics will be.
- Pre-Experimentation Setup – After you set up the hypothesis, you need to determine your sample size in order to detect a statistically significant change in your results.
- Experiment Implementation – Configure the filter with the ability to separate your audience into groups, and keep their treatments separated through the duration of the experiment.
- Execute Experiment – Launch the pages and wait. Wait until you get sufficient sample size before drawing conclusions. Guard against publishing premature results.
- Evaluate Experiment – Now that the experiment is over, put the results into your favorite statistics tool and evaluate the difference between treatments. If the groups are dimensionalized in a web analytics platform, you may also want to look at additional cross-effects with other data.
- Summarize Results – Report the results (even if the learning was not as planned)
- Take Action – Make production-level changes, if necessary.
- Monitor Actions – Validation, Control, and Dashboard metrics.
- Exploratory Data Analysis – What do we want to test next…?
Prototyping
Sometimes we simply cannot wait to set up and run a controlled experiment. Prototyping several low-fidelity models can cut your development time quickly and build a fact-base of feedback in minutes, rather than days or weeks.
Author Tom Kelley cites two guiding principles for effective prototyping: Real User Feedback and Multiple Options.
Feedback will be more valid and convincing if received from someone that has a personal stake in the process—getting feedback from a set of “typical” customers, and not just from your roommate who dreams in code—will likely yield results that are better for your sales. For best results, use as many real customers as you can.
Presenting multiple options to real stakeholders—actual users of the site—takes your personal attachment out of any one option. Users will tend to give more reserved feedback if they feel they are being guided toward a specific conclusion, as when being asked to comment on a single option. Conversely, multiple prototypes can often inspire users to be more open with their feedback and even suggest additional options by combining the best elements that each design has to offer.
Having an Experimenter on your team pushes everyone to think creatively, and the discipline of framing success metrics as hypotheses brings rigor and critical thinking. It can also be a great source of fun for the team – even redefining what it means to “fail wisely,” and perhaps most important, to dispel the fear of taking the first step – asking WHY?
Concepts in this series are based on the book, The Ten Faces of Innovation, by Tom Kelley, 2007.
Comments