Let A/B Tests do the business
As product manager, I have been reading a bunch of articles on A/B testing to find answers to what, why, when and how of A/B Tests to summarize my understanding below -
AB testing is nothing but a business decision based on statistically discovered insights. AB testing not only helps in understanding of what works for your users but also gives a better understanding of user behaviors and patterns. In other words, it’s not about finding a fruit but discovering an orchard and knowing when to which fruits in it.
What is A/B Testing — A/B testing, or split testing, is a process of running a simultaneous experiment between two options to identify which option would results in maximizing desired output from the users.
There are three types of A/B tests:`
A/B Split Testing — Basic test validating one element through two variations at a time
Multivariate Testing — Testing multiple elements simultaneously. To identify the right combination of elements to maximize the conversion
Experimental Design — Developing your own research method for an in-depth analysis of a specific element
Why A/B Testing is important-A/B testing will not only provide insights on what works with users but also on the behavior of the users. Important point to note here is what works with one type of user many not work for other type of user. Also, what worked for a user in winter may not work for the same user in summer. A/B tests can provide crucial data on the users (gender / age/ location etc.), time of the test (to identify user’s dictions and influence of time) and user journey flow (in the user flow, which type of user is dropping at what stage in the conversion flow).
The real challenge is not to get traffic to the website but to convert that traffic to accomplish business goals to justify huge expense on acquiring of this traffic.
Why carry out A/B Tests- Why not !
With a lot of free / may be paid tools A/B testing has become very easy and pocket friendly to suit every type of organization. Also, it will not interrupt the user. It is the best way of finding what works for a particular user without really impacting the running business.
When is it done-A/B tests are carried out throughout the product lifecycle. Three most common objectives of A/B tests are to enhance site navigation, validate any change in the website and to increase call to action clicks.
How is it done- Below is a basic framework to carry out AB tests:
Define Business Goals — First step is to understand what is the main purpose of your website and what are you trying to achieve from it. E.g. for a social networking site — number of new signups. For an eCommerce website number of online sales
Understand Data — There are many ways to gather traffic data. Pathway analysis and funnel analysis are common methods of gathering data. More than just gathering data it is important to understand the data
Prioritize User Problems — The data will uncover a lot of user problems however the priority goes to the problems which are related to defined business goals
Hypothesis — This is basically a problem which will be solved by A/B tests. E.g. 35% of the first time users do not sign up for the networking site because they are not sure if their friends are on it or not. So if we could show profiles of their friends on the sign up page, it would increase the sign ups
Prioritize Hypothesis — ICE (impact/ confidence/ ease) is a widely used framework to prioritize the hypothesis. Based on the business goal and impact any customized framework be used here.
Learn & improve- See result, validate success then either move to next hypothesis or iterate the current one.
Conclusion
A well-defined business goal can be quantified and could be represented by a website conversion. A/B test can validate the hypothetical correlation between the business goal and selected website conversion rate. A/B test done right is statistical way of understanding the users and offering them exactly what they want. These valuable insights can help completing desired business goals.
Link to my Linked Post : https://www.linkedin.com/pulse/let-ab-tests-talk-business-2018-sarvadnya-arjunwadkar/
