A/B Testing

A/B testing allows you to compare different versions of your web content to see which one performs better. It's a method that involves showing up to five variants to similar visitors at the same time, to determine which one is more engaging.

In the context of our application, A/B testing is used to compare different versions of article elements such as titles, images, and descriptions. The goal is to determine which variant generates more clicks and, therefore, more audience engagement. When a new experiment is created, a variant of the title or image is injected into the newspaper page. This variant is shown to 50% of the users, while the original is shown to the other 50%.

The following sections will provide an overview of the winner selection strategies we use in our A/B testing process.

To start an A/B test, follow these steps:

1. Access the extension: Open the Publisher Analytics extension.

2. Select articles tab: Click on the “Articles” tab within the extension. This will display a list of all your articles.

3. Choose an article: Browse the list of articles and select the one you wish to conduct an A/B test on. Clicking on your chosen article will trigger a modal to appear.

4. View original variant: In the modal that appears, you'll see the original variant of the article on the left side. This is the current version of the article that users are seeing.

5. Add test variants: To add a new variant you want to test against the original, click on the plus sign located on the left side of the modal. You can add up to five different variants for testing. For each variant, you have the option to add or change images, titles, and descriptions. It's not mandatory to alter all these elements for every variant; you can choose to test just one aspect, like the title or the image, by leaving the other fields blank.

6. Utilize AI suggestions: If you’re unsure what text to use for a title or description, you can take advantage of the "AI suggestion" button. This feature automatically generates text for you. You can then review and edit this text to better suit your needs.

7. Monitor experiment progress: Throughout the A/B testing period, you can track the performance of each variant. Click on the variants to see detailed metrics such as quality clicks, low-quality clicks, and scroll depth. This data will help you understand which variant is performing best.

8. Conclude the test: Once the experiment is concluded, a button will be available to automatically implement the winning variant for all users moving forward. However, if you prefer manual control, you can also update the winning variant through your Content Management System (CSM).

9. Experiment deletion: If you decide that an experiment is no longer needed, you can easily delete it. Look for the delete button within the A/B testing interface to remove the experiment from your list.

By following these steps, you can efficiently conduct A/B tests on your articles, ultimately improving user engagement and optimizing your content based on empirical data.

Winner Selection Strategies

Time-Bound Interaction Strategy

This strategy ensures that each variant is presented to users an equal number of times until a specified duration is reached. The winner is determined by the total number of clicks each variant accumulates within this time frame.

How It Works:

1. Equal Exposure: Each variant is randomly displayed to users, maintaining equal exposure over the test period.
2. Data Collection: Clicks for each variant are tracked until the end of the specified time period.
3. Winner Determination: The variant with the highest total number of quality clicks at the end of the time period is declared the winner.
4. Tie-Breaking: In case of a tie on the primary metric (quality clicks), secondary metrics such as average scroll depth and total variant clicks are considered in a hierarchical manner to break the tie.
5. Fallback: If no clear winner is identified based on the metrics (e.g., all metrics are zero), the predefined original variant is selected as the default.

This strategy is ideal for tests where a fixed duration is preferred, ensuring that each variant has an equal chance to prove its effectiveness within the set timeframe.

Tie-Breaking Logic:

When variants have the same number of quality clicks, the tie is resolved by sequentially considering additional metrics. The process is as follows:

1. Primary Metric (Total Variant Clicks): The variant with the highest number of total clicks is initially considered the winner.
2. Secondary Metric (Quality Clicks): If there is a tie on total clicks, the variant with the higher number of quality clicks takes precedence.
3. Tertiary Metric (Average Scroll Depth): If the tie persists, the average scroll depth percentage for each variant is the final deciding factor.
4. Controllable Variant Selection: If all compared metrics are equal (including the scenario where all metrics are zero), the system defaults to the original variant, which is a predefined safe option.

This tie-breaking logic ensures a systematic approach to selecting a winner when the primary metric does not yield a decisive result.

Thompson Sampling Strategy

Thompson Sampling is a Bayesian method that dynamically adjusts the frequency of each variant's presentation based on ongoing performance data. This method allows for early termination of the test if a variant reaches a high confidence level of being the best option, optimising for performance and efficiency.

How It Works:

1. Initial Distribution: Variants are initially presented randomly to users.
2. Performance Update: The algorithm updates the probability of each variant being the best option based on live click data.
3. Probabilistic Selection: Variants with higher success probabilities are shown more often to users.
4. Continuous Learning: The algorithm learns and adapts the distribution of variants in real-time.
5. Early Termination: If a variant achieves a confidence level above a certain threshold, the test can conclude early, declaring the high-performing variant as the winner.
6. Maximum Duration: If the test does not reach a conclusive result through early termination, it will continue until a pre-set maximum duration is reached. At this point, the algorithm will select the variant with the highest estimated probability of being the best, based on the Bayesian model's current probability distribution. In cases where two or more variants have similar performance data, the selection between them is influenced by the underlying probabilistic model, which means that either variant could be chosen. The final decision is made with the best available information, taking into account the uncertainty that is integral to the model.

Thompson Sampling is advantageous for tests where you want to identify the best-performing variant quickly and reduce the exposure to less effective options, potentially improving the overall user experience during the test.