A/B testing is a valuable tool for discerning the effectiveness of new features on an online shop, such as the Fit Quiz.
While we don’t provide our own A/B testing tools, we can assist with the setup of an A/B test in a testing tool you’re using.
To ensure a successful A/B test of the Fit Quiz, follow this guide which outlines best practices.
What to do before starting the test
Make sure that you have completed the following steps before starting the test:
Fit Quiz has been configured and customized to your desired level
Fit Quiz is fully integrated, whether through our Shopify app, library, or API. This also applies to order and return tracking
Our team is aware of the test, preferably 24-48 hours in advance
Limitations of simple split tests
Traditional A/B tests are designed to test simple, static elements on a page, such as different versions of a call-to-action button or static notifications.
However, Fit Quiz functions differently. While the Fit Quiz call-to-action may appear on a page, not every shopper will engage with it and/or receive a recommendation.
Therefore, it's important to analyze the full funnels (shoppers that engaged, purchased, or returned the recommended size) separately to avoid drawing incorrect conclusions.
Coordinate and Communicate
We recommend coordinating your A/B testing efforts with our team to get additional tracking from our side and ensure a statistically sound test. After the launch, both teams should exchange relevant information, especially during the first days of the test to confirm that all the tracking is working correctly from both sides.
Some crucial metrics, like the exact recommended size, are only tracked on our end. Using these metrics is an important step in assessing the overall results of tests. Please reach out to us to gain access to the metrics and coordinate testing efforts.
Duration and Scope of the test
We suggest running the test for a minimum of 3 months (preferably 4-6 months) to collect enough data on the impact of the Fit Quiz and reach statistical significance of the tests.
The test can be split into various stages, for example,
1-2% of the traffic for a week to ensure the technical setup and tracking are sound
5-10% for 2-4 weeks to analyze user engagement and potentially revise the UI/UX
following by 10-15% (or higher) for the remainder of the test to test the final version of Fit Quiz
Additional criteria
Product categories | Tests will run across all the relevant categories, unless specified and requested by you. It's also possible to limit the test to specific geographies, websites, product categories or even specific products. |
Allocated traffic | All the traffic on these categories shall be assigned to an A/B test:
We suggest at least 10-15% (ideally 20%) of the traffic to be assigned to Group B. Allocation of the traffic can be done gradually: for example, starting with 1-2% and gradually increasing to the desired %. |
Product coverage by Fit Quiz | Within the allocated group Fit Quiz aims to cover 100% (or close to 100%) of products. |
Analyzing results and metrics
The ultimate goal of the A/B test is to prove that the Fit Quiz can have a significant positive impact on returns and sales at scale.
Here're metrics that we recommend to track and analyze:
How much do shoppers engage with Fit Quiz? |
|
Click rate on Fit Quiz call-to-action button from PDP |
|
Completion rate: % of shoppers who opened Fit Quiz and got the recommendation (e.g. completed all the questions) |
|
Do shoppers purchase more when using Fit Quiz? |
|
Sales conversion |
|
Share of orders with the recommendation |
|
Share of Gross Sales/GMV with the recommendation |
|
Do shoppers return less when using Fit Quiz? |
|
Return rate (overall returns) |
|
Return rate (size and fit-related returns) |
|
Additional return metrics can be analysed as well: returns by category/sub-category, returns value (in USD, EUR etc.) |
|
Ensuring Statistical Significance
In addition to confirming that the A/B test reached statistical significance, the following metrics should serve as "control metrics":
The click rate of Fit Quiz CTA should be close to or higher than the Size Chart CTA
The sales conversion of Group A and Group B that didn’t engage with Fit Quiz should be similar
The return rate for Group A and Group B that didn’t engage with Fit Quiz should be similar
Following these guidelines will help ensure that your A/B testing of the Fit Quiz is productive and provides meaningful, actionable results.
If you require any additional help and assistance with setting up an A/B test, please reach out to us and we will be happy to assist.