The rise of experimental evaluations within organisations has the potential to transform organisational decision-making. Yet while there has been a rapid growth in experiments, too many are run incorrectly. Here are seven steps to ensure that your experiment delivers.
1. Identify a narrow question.
Your question should be testable. Instead of asking if advertising is worth the cost, ask “How much does advertising our brand name on Google AdWords increase monthly sales?” This is an empirical question that an experiment can answer.
2. Use a big hammer.
It may sound appealing to start small in order to avoid disrupting things. But your goal should be to see whether some version of your intervention — your new change — will make a difference to your customers, which requires a large intervention.
3. Perform a data audit.
Make a list of all of the internal data related to the outcome you would like to influence and when you will need to do the measurements. Include data both about things you hope will change and things you hope won’t change as a result of the intervention.
4. Choose a study population.
Choose a subgroup among your customers that matches the customer profile you are looking to understand. But beware: If your subgroup is not a good representation of your target customers, the findings of your experiment may not be applicable.
Randomly assign some people to a treatment group and others to a control group. The treatment group receives the change you want to test, while the control group receives what you previously had to offer. The first rule of randomisation is to not let participants decide which group to be in, or the results will be meaningless. The second is to make sure there really are no differences between treatment and control.
6. Commit to a plan, and stick to it.
Before you run an experiment, lay out your plans in detail. How many observations will you collect? How long will you let the experiment run? What variables will be collected and analysed? Record these details.
7. Let the data speak.
To give a complete picture of your results, report multiple outcomes. Sure, some might be unchanged, unimpressive or inexplicable. But better to be transparent about them than to ignore them.