Design Thinking — Getting customers into Perspective
The web provides us with an opportunity to evaluate ideas quickly using:
- Controlled experiments
- Randomized experiments
What is a controlled experiment?
“Controlled,” in this sense, means putting restrictions in place so that certain variables don’t impact the outcome of your experiment — means putting restrictions in place so that certain variables don’t impact the outcome of your experiment.
What is a randomized experiment?
It basically means doing the best you can, but it will also cost you a lot of money to do, and you won’t make any money in the process. So you have to decide how much randomizing is worth the cost.
What is a randomized controlled experiment?
When people hear the term, they most often think of clinical trials, where one group is given a treatment and another a placebo — they just need to be “controlled” and include an element of “randomization” as explained above.
In this, the users are randomly assigned to one of two variants:
(i) Control— Existing, which is the current scenario.
(ii) Randomized — Treatment, which usually is a new version under evaluation.
Metrics of interest are collected and statistical tests are then conducted over the collected data to evaluate whether there is a statistically significant. The difference between the two variants on metrics of interest thus permits us to retain or reject the Null Hypothesis (i.e. no difference between the versions)
In many cases, drilling down using machine learning or data mining techniques allows us to understand which sub-populations show significant differences, thus helps improve our understanding and progress forward with an idea.
Conversions:
The conversion rate of an e-commerce site is the percentage of visits to the website that include a purchase.
If a designer show you few UI’s of a same element and asked which one should be deployed?
(i) Could you tell which one results in a higher conversion rate?
(ii) Could you estimate what the difference is between the conversion rates and whether that difference is significant?
Example:
Let’s say you add a coupon code box as a new feature in the payment UI page which was not present earlier.
It may happen that the company may loose majority of their revenue. Although, the changes in the upgrade was positive, but here the coupon code is a critical part, people might start to think twice about whether they were paying too much because there are discount coupons out there that you do not have.
How we can measure the results of change?
Experiments may have multiple objectives and a scorecard approach might be useful, although selecting a single metric, possibly as a weighted combination of such objectives is highly desired and recommended.
A single metric forces trade offs to be made once for multiple experiments and aligns the organization behind a clear objective.
A good metric should not be short-term focused (e.g., clicks), to the contrary, it should include factors that predict long-term goals, such as predicted lifetime value and repeat visits.
Evaluation:
To evaluate whether one of the treatments(randomization) is different than then control, a statistical test can be done. We accept a treatment as being statistically significantly different if the test rejects the null hypothesis.
Limitations:
Despite significant advantages that controlled experiments provide in terms of causality, they do have limitations that need to be understood.
Quantitative Metrics, but No Explanations: It is possible to know which variant is better, and by how much, but not ―why.
Short term vs. Long Term Effects: Controlled experiments measure the effect on the overall metric during the experimentation period, typically a few weeks. While some people have criticized that focusing on a metric implies short-term focus.
Primacy and Newness Effects: These are opposite effects that need to be recognized. If you change the navigation on a web site, experienced users may be less efficient until they get used to the new navigation, thus giving an inherent advantage to the Control. Conversely, when a new design or feature is introduced, some users will investigate it, click everywhere, and thus introduce a ―newness‖ bias.
Launch Events and Media Announcements: If there is a big announcement made about a new feature, such that the feature is announced to the media, all users need to see it.
Implementation of the system
Implementing an experiment on a website involves two components.
- The first component is the randomization algorithm, which is a function that maps users to variants.
- The second component is the assignment method, which uses the output of the randomization algorithm to determine the experience that each user will see on the website.
Finding a good randomization algorithm is critical because the statistics of controlled experiments assume that each variant of an experiment has a random sample of users.
Properties of algorithm:
(i) Users must be equally likely to see each variant of an experiment (assuming a 50–50 split). There should be no bias toward any particular variant.
(ii) Repeat assignments of a single user must be consistent; the user should be assigned to the same variant on each successive visit to the site.
(iii) When multiple experiments are run concurrently, there must be no correlation between experiments i.e user’s assignment to a variant in one experiment must have no effect on the probability of being assigned to a variant in any other experiment.
(iv) The algorithm should support monotonic ramp-up i.e the percentage of users who see a Treatment can be slowly increased without changing the assignments of users who were already previously assigned to that Treatment.
Conclusion
- Adopting the practices above we can not only mitigate the intuition based decision making of an individual but can thoroughly investigate the results.
- This gives us the seamless power to understand our customer base and hence a detailed statistics of what will and is currently work for us!
Vivek Gupta: https://www.linkedin.com/in/vivekg-/
Follow me on Quora: https://www.quora.com/profile/Vivek-Gupta-1493
Check out my legal space here: https://easylaw.quora.com
