Skip to main content
All CollectionsBasics
How to efficiently run an A/B test of Fit Quiz
How to efficiently run an A/B test of Fit Quiz

Thinking of running an A/B test to assess Fit Quiz's impact on your shop? Use this guide to set it up and correctly analyze results

Updated over 8 months ago

A/B testing is a valuable tool for discerning the effectiveness of new features on an online shop, such as the Fit Quiz.

While we don’t provide our own A/B testing tools, we can assist with the setup of an A/B test in a testing tool you’re using.

To ensure a successful A/B test of the Fit Quiz, follow this guide which outlines best practices.

What to do before starting the test

Make sure that you have completed the following steps before starting the test:

  • Fit Quiz has been configured and customized to your desired level

  • Fit Quiz is fully integrated, whether through our Shopify app, library, or API. This also applies to order and return tracking

  • Our team is aware of the test, preferably 24-48 hours in advance

Limitations of simple split tests

Traditional A/B tests are designed to test simple, static elements on a page, such as different versions of a call-to-action button or static notifications.

However, Fit Quiz functions differently. While the Fit Quiz call-to-action may appear on a page, not every shopper will engage with it and/or receive a recommendation.

Therefore, it's important to analyze the full funnels (shoppers that engaged, purchased, or returned the recommended size) separately to avoid drawing incorrect conclusions.

Coordinate and Communicate

We recommend coordinating your A/B testing efforts with our team to get additional tracking from our side and ensure a statistically sound test. After the launch, both teams should exchange relevant information, especially during the first days of the test to confirm that all the tracking is working correctly from both sides.

Some crucial metrics, like the exact recommended size, are only tracked on our end. Using these metrics is an important step in assessing the overall results of tests. Please reach out to us to gain access to the metrics and coordinate testing efforts.

Duration and Scope of the test

We suggest running the test for a minimum of 3 months (preferably 4-6 months) to collect enough data on the impact of the Fit Quiz and reach statistical significance of the tests.

The test can be split into various stages, for example,

  • 1-2% of the traffic for a week to ensure the technical setup and tracking are sound

  • 5-10% for 2-4 weeks to analyze user engagement and potentially revise the UI/UX

  • following by 10-15% (or higher) for the remainder of the test to test the final version of Fit Quiz

Additional criteria

Product categories

Tests will run across all the relevant categories, unless specified and requested by you. It's also possible to limit the test to specific geographies, websites, product categories or even specific products.

Allocated traffic

All the traffic on these categories shall be assigned to an A/B test:

  • Group A (without Fit Quiz)

  • Group B (with Fit Quiz)

We suggest at least 10-15% (ideally 20%) of the traffic to be assigned to Group B. Allocation of the traffic can be done gradually: for example, starting with 1-2% and gradually increasing to the desired %.

Product coverage by Fit Quiz

Within the allocated group Fit Quiz aims to cover 100% (or close to 100%) of products.

Analyzing results and metrics

The ultimate goal of the A/B test is to prove that the Fit Quiz can have a significant positive impact on returns and sales at scale.

Here're metrics that we recommend to track and analyze:

How much do shoppers engage with Fit Quiz?

Click rate on Fit Quiz call-to-action button from PDP

  • may be subject to testing and revisions

  • desired figure: no lower that the click rate of the Size chart

Completion rate: % of shoppers who opened Fit Quiz and got the recommendation (e.g. completed all the questions)

  • may be subject to testing and revisions

Do shoppers purchase more when using Fit Quiz?

Sales conversion

  • calculated specifically for shoppers who have engaged with Fit Quiz

  • compared to shoppers in Group A and shoppers in Group B who didn’t engage with Fit Quiz

Share of orders with the recommendation

  • calculated for Group B

  • shall only include cases when a shopper purchased the recommended size (without changing it)

Share of Gross Sales/GMV with the recommendation

  • calculated for Group B

  • shall only include cases when a shopper purchased the recommended size (without changing it)

Do shoppers return less when using Fit Quiz?

Return rate (overall returns)

  • calculated specifically for shoppers who have engaged with Fit Quiz

  • compared to:

    • shoppers in Group A

    • shoppers in Group B who didn’t engage with Fit Quiz

    • shoppers in Group B who purchased a size different from the recommendations → this is a true indicator of the accuracy of the recommendation regardless of shopper’s prior knowledge of a brand/product (shoppers who likely won’t use Fit Quiz)

Return rate (size and fit-related returns)

  • calculated specifically for shoppers who have engaged with Fit Quiz

  • compared to:

    • shoppers in Group A

    • shoppers in Group B who didn’t engage with Fit Quiz

    • shoppers in Group B who purchased a size different from the recommendations → this is a true indicator of the accuracy of the recommendation regardless of shopper’s prior knowledge of a brand/product (shoppers who likely won’t use Fit Quiz)

Additional return metrics can be analysed as well: returns by category/sub-category, returns value (in USD, EUR etc.)

Ensuring Statistical Significance

In addition to confirming that the A/B test reached statistical significance, the following metrics should serve as "control metrics":

The click rate of Fit Quiz CTA should be close to or higher than the Size Chart CTA

The sales conversion of Group A and Group B that didn’t engage with Fit Quiz should be similar

The return rate for Group A and Group B that didn’t engage with Fit Quiz should be similar

Following these guidelines will help ensure that your A/B testing of the Fit Quiz is productive and provides meaningful, actionable results.

If you require any additional help and assistance with setting up an A/B test, please reach out to us and we will be happy to assist.

Did this answer your question?