Distinctive Analytics

A/B tests are randomly controlled experiments. In A/B testing, you get user response on various versions of the product, and users are split within multiple versions of the product to figure out the “winner” of the version.

Users of a service, software, app, or website are subjected to A/B testing. There are also multiple instances of A/B testing in the making of Hollywood movies. Where the multiple endings to one title were shot and shown to different audiences. And the better-performing one was rolled out. (Man of Steel, I am legend, the butterfly effect)

We can apply hypothesis testing in A/B testing to find the better-performing version of the product. We can formulate the hypothesis as one version being null and the other being alternate. The assumption behind a null hypothesis stands for whether the assumption is more likely to occur or not.

Example 1:

Consider a webpage as an example. It normally gets five minutes of stickiness on average. 

Furthermore, we decided to make some changes (like introducing banners informing about discounts on products) in order to increase user stickiness. After the changes, user stickiness increased by 7–8 minutes on average.

Null Hypothesis

H0: User stickiness is the same after the change.

Alternate hypothesis

H_a: User stickiness increased after the change.

Example 2: (in depth)

Let’s take another example of an edtech app.

This app is having problems because only 30% of visitors are getting retained until next week, when it needs to be more than that.

Steps to perform tests

1. Data gathering

The first step would be to collect data regarding the problem.

In this case, we need data

Narrow down the root of the problem.

The problem seems to be the journey after the quiz’s start page. Visitors are leaving the site after this page. There may be multiple issues here. Perhaps a lack of interest from the home page to quizzes, or perhaps low-quality content. 

2. Create alternate version

Following the research, we will be able to create multiple versions of the app, one of which will contain high-quality content.

On the other hand, other version with the smoothest navigation and discovery experience.

Now on both these versions, we can examine how our KPIs are performing for a considerable time, maybe for a week or two, and it will tell us the entire story.

3. Run the alternate version

    Run your alternate versions of the app. Make sure you have separate audiences for all versions so that you can collect randomized data.

Perform hypothesis tests

    In some cases, we can run simple visualizations to determine the result.

Always check if your version has a statistically significant outcome over the older version. We can run hypothesis testing now to get the statistically significant value.

At this point, we might have a winner. But it is best practice to validate the gathered test data or run the test again against other versions of the app.

From this use case, you get the idea of how to perform the A/B tests. In an actual use case, there are more details to work on.

Example 3:

A/B tests with live user input are now used by many e-commerce websites. E-commerce websites use A/B testing to compare a product’s variations, or certain products that are the same but have different features highlighted.

As an example, consider the wholesale laptops and accessories industry. It sells laptops in bulk. And its marketing strategy mostly focuses on spreading networks to showrooms and dedicated laptop shops. After a quarter, their GMV (gross merchandise value) does not meet the expected target for that quarter. Following some research, it was determined that this was the incorrect sales strategy in terms of audience.

Null Hypothesis:

Sales strategy is not the cause for low GMV

Alternate Hypothesis:

The wrong sales strategy is affecting GMV.

Normal Version:

The company has set up a website where it displays each laptop for purchase in quantities of 100 or 200.

Alternate Version:

So they decided to rethink their sales strategy. As their focus is B2C sellers (Dedicated Laptop Shops), they might require more variety in products than bulk.

As a result, the website lists units ranging from 10-20-50 per unit for all types of laptops and accessories sold as a showroom package.

Data Gathering

The audiences should be divided between the alternate version and the old version for the first half of the business quarter. And the sales data was gathered.

Sales rose by 25% with the alternate version. In matters of business, an alternate version is a winner. But statistically, it doesn’t always come out as a win. It matters when there is such a gap between statistical proof and real-world proof.

In the above use case, we could have also made an alternate version by changing the whole audience. But making some changes inward rather than outward makes a financially efficient decision. Changing the PR strategy, Redefining networking on the field, necessitates more funds.