What do marketing and drug trials have in common?

Recently, I read a research paper entitled "Bayesian Statistics and the Efficiency and Ethics of Clinical Trials".
The article asks a few interesting questions about clinical drug trials. As I pondered, I discovered a correlation between trails and website / marketing experimentation.
Let me explain:
Clinical trials are experiments conducted during clinical research; such as pharmaceutical drug research.
When a company develops a new drug to benefit to society, one step in the process is to actually test the drug.
It's test subjects are animals, such as mice and monkeys.
If all goes well, human clinical trails begin to see if how the drugs work.
But, "what exactly is a clinical trail you ask?"
Good question!
Models, simulation, and guesses only go so far.
At some point, someone has to actually try the new drug. This is called a clinical trial.
But these trials can raise some ethical questions. Is the purpose of a clinical trial to help patients or to advance research?
Both, would serve as the optimal option. But, when these two purposes start to be at odds with each other, what do you do?
Imagine you are a cardiologist (a heart doctor) and you have an experimental drug to try.
You have 100 patients that you divide into two groups. One group will receive the standard treatment, and one group will take the new drug.
You know that the standard treatment works about 50% of the time.
As you bring on patients into the new drug group, it starts to look like the success rate is much lower than 50%.
Half way through the test you suspect the new treatment is not helping your patients. Now what do you do?
The standard treatment appears to improve the chance of the remaining patients to get better.
You know, there are strict mathematical rules when it comes to statistical significance in experimentation.
Yet, stopping the experiment early, would remove any statistical certainty from the experiment.
So what do you do?
Do you act as a good researcher or a good doctor?
Do you help your patients or do you advance scientific research?
It is a hard question.
Let’s take it out of life and death and move it to an easier arena… Marketing.
Similar trials can also done in marketing.
We can divide our audience in half, send one group a tried and true campaign, and send the other a new campaign.
If both campaigns are trying to sell the same product, we can measure how many we sell from each and know which campaign is better.
But what if half way through the experiment it starts to look like the new campaign isn’t selling as much as the original campaign?
Do you stop the new campaign and send everyone back to the original?
You’ll make more money that way but you’ll sacrifice any statistical significance in the experiment.
What do you do? Do you be a good researcher or do you be a good marketer?
It’s the same question and it’s as hard here – even though the stakes aren’t as high.
The paper offers has a few solutions to this problem in the clinical trial world that we can apply to our marketing world:
1. Create a Data Monitoring Committee who is separate from the researchers the are performing the tests. This puts some separation between the patients and the decisions which makes the hard decisions a little easier. For most marketing situations, this is overkill.
2. Create a set of auxiliary endpoints. Most experiments have one main goal that they are trying to maximize but there are often side effects that can be measured too. For example, you may choose product sales as the goal of your experiment but you also care about the bounce rate as well. You can set up some extra endpoints for the experiment that trigger the experiment to stop early if one of these side effects starts to go out of a specified range.
3. Use a Bayesian Statistics model for the experiment instead of a standard experiment. The details are complicated but the difference can be explained. In a standard experiment, you divide your sample into two groups and run each group through the experiment. After all the groups have been tested, you see which one had more successes and that is considered the winner. This model leads to the moral problems that I mentioned above. On the other hand, instead of waiting until the end, a Bayesian model will continue to measure the probability that one group is better than the other. At the beginning, the groups are 50/50 – they both have an equal chance. As you start to run tests, those probabilities start to change. After a while, they may be 70/30. This doesn’t mean that it’s 70% better but that there is a 70% chance that it is better. There is a 30% chance that it’s the same or worse. If one of the groups is actually better, after a while the probabilities will continue to drift towards that one. This method lets you stop the experiment at any time and know the probability that one of the groups is better than the other.
As you can see, the connection between clinical drug trials and your website is not that big of a stretch. I knew this when I wrote Title Experiments and that is why I decided to use Bayesian Statistics at the heart of the plugin. Not only is it used to save lives but it helps you write better titles for your website as well. I call that a win win scenario!