At a Glance
- HelloFresh runs large numbers of concurrent A/B, ABC, and ABCD tests, creating heavy Bayesian inference workloads.
- The original MCMC-based pipeline suffered from inefficient sampling with strong parameter correlations and high autocorrelation in the chains.
- A redesigned Bayesian model with one fewer parameter and revised priors fixed sampling issues and unified A/B, ABC, and ABCD tests.
- Parameter recovery simulations confirmed that the new models produce accurate and trustworthy inferences on conversion probabilities.
- Refactoring the computation into one large unpooled model to fit all datasets simultaneously reduced batch runtime from 5–6 hours to 5–6 minutes.
- The new approach makes the pipeline scalable, reduces compute demands, and enables much faster decision making for HelloFresh.
What Problem Did HelloFresh Face With Bayesian A/B Testing?
HelloFresh uses A/B testing extensively to decide which of several options leads to higher conversion probabilities. A conversion, or success, can be any event of interest such as a click, sign‑up, or purchase. Customers are randomly exposed to alternatives A or B (or more options such as A/B/C or A/B/C/D), and conversions are recorded over time so that inferences can be made about the underlying conversion probabilities.
The practical challenge is deciding when there is sufficient evidence to stop a test and adopt the best‑performing option. Companies do not want to keep showing inferior variants once there is clear evidence that another option performs better. For HelloFresh, the immediate request was different: they needed to make their Bayesian A/B testing pipeline efficient and scalable.
A company at HelloFresh’s scale runs a large number of A/B, ABC, and ABCD test campaigns simultaneously. Tests start at different times and take varying durations to reach a decision threshold. There is also interest in tracking how conversion rates evolve over time. Together, this leads to thousands of tests and significant compute costs.
While Bayesian methods for standard A/B tests have received attention, implementations for ABC and ABCD tests have been explored less. HelloFresh’s data scientists had working Bayesian models, but MCMC sampling efficiency was low. Strong parameter correlations and high chain autocorrelation meant that compute resources were not being translated efficiently into information about conversion probabilities. For an organisation making many bottom‑line‑critical decisions, this lack of efficiency and reliability was unacceptable.
Because of the raw number of tests and the sampling problems, a typical overnight batch of analyses took around 5–6 hours of compute time. With the number of tests only expected to grow, this runtime was not scalable and limited how quickly teams could get feedback and make decisions.
How Was Bayesian Inference Improved for ABC and ABCD Tests?
The first step was to address the inference issues before attempting major speed optimisations. Establishing a correct Bayesian model with efficient sampling is essential; in many cases, fixing the model can also yield speed improvements. Diagnosing the existing models revealed significant correlation between some parameters in the posterior and too much autocorrelation in the MCMC chains. These issues both slow down convergence and reduce the effective sample size for a given compute budget.
After investigation and model exploration, a structurally different Bayesian model was proposed. The new model:
- Removed one parameter compared to the original formulation.
- Used altered priors tailored to the problem.
- Applied the same structure consistently to A/B, ABC, and ABCD tests.
This new specification solved the inefficient sampling problems. It also provided intuitively appealing priors on conversion probabilities that aligned with HelloFresh’s domain knowledge. For example, prior samples for an ABC test produced conversion rate vectors ((p_1, p_2, p_3)) where:
The prior placed high probability on conversion rates being similar across variants rather than wildly different. Prior expectations over the possible range of conversion rates between 0 and 1 were approximately uniform.
With the new model, MCMC traces showed well‑behaved sampling. Visual inspection of the chains was “very pleasing,” and checks of parameter correlations and autocorrelation (not shown) confirmed that the earlier sampling issues were resolved.
To verify that the new models were not only efficient but also accurate, parameter recovery simulations were run. These simulations used known underlying conversion probabilities and simulated data to check whether the posterior distributions recovered the true values. For an ABC test, posterior distributions were correctly centred on the true conversion probabilities, as indicated by plot markers aligning with crosshairs representing the ground truth. This step provided crucial reassurance that:
- The revised models were estimating meaningful quantities.
- The inferences could be trusted for real decision making.
At this stage, the improved models already yielded speed gains:
- Around 1.2x faster for A/B tests.
- Roughly 2x faster for ABC and ABCD tests.
These were worthwhile improvements, but the goal at PyMC Labs was more ambitious, so work continued.
How Was the Inference Time and Compute Demand Reduced?
Were Simple Tuning and Recompilation Changes Enough?
Experimentation showed that MCMC chains for the new models converged after only a handful of steps. This suggested that reducing the number of tuning steps might provide a speed gain. The first optimisation attempt reduced the default 1000 tuning steps down to 100. However, the effect was small, with only about 0.1 seconds saved per A/B test. Given the scale of HelloFresh’s batch runs, this marginal improvement was not sufficient.
The next idea was to avoid recompiling the same PyMC model repeatedly. Instead of building a fresh model object for each dataset, a single model could be defined once, with data handled via pm.Data. New datasets could then be fed into this fixed model by switching the pm.Data containers. While this avoided repeated compilation and improved the engineering design, the resulting speed increase in practice was negligible. More dramatic changes were needed.
How Did the Unpooled Model Deliver Massive Speedups?
The breakthrough came from reframing how the tests were fit rather than just how each individual fit was configured. Instead of iterating over datasets and running MCMC separately for each test, a single large unpooled model was constructed to fit all datasets simultaneously. In an unpooled model of this kind:
- Each test (e.g. each A/B campaign) has its own parameters.
- These sets of parameters are statistically independent within the model.
- The structure is identical across tests, but their parameters do not borrow strength from one another.
Conceptually, this is equivalent to running many identical Bayesian A/B inferences in one large joint model. Because the tests are independent, the unpooled model simply groups them in a single inference pass instead of many separate passes. For A/B tests, the model was adapted so that:
- One PyMC model represented all tests at once.
- Each dataset corresponded to one independent set of conversion probability parameters within that global model.
By doing this, the pipeline no longer needed to loop over hundreds or thousands of small models. Instead, PyMC compiled and sampled one model that encoded all tests at once. This change traded a large number of small independent sampling runs for a single, more substantial run with many independent components. In practice, this was far more efficient for the same total amount of data and number of parameters.
What Speedups Did HelloFresh Achieve?
Initially, HelloFresh’s batch A/B testing pipeline took around 5–6 hours to run overnight, given the number of concurrent tests and the original model and sampling setup. With the new model structure, improved priors, and the unpooled multi‑dataset approach, the total time dropped to about 5–6 minutes. This represents a roughly 60x speedup for the full batch pipeline. The outcome transformed how HelloFresh could use Bayesian A/B testing:
- Much faster decision making: Teams can reach conclusions from tests far more quickly, without waiting hours for batch jobs to finish.
- Reduced compute costs: Expensive compute resources are no longer required to run the pipeline at scale.
- Future‑proof scalability: As the number of tests and volume of data grows, the redesigned pipeline is equipped to handle the workload.
How Does This Work Relate to A/B, ABC, and ABCD Tests?
The redesigned Bayesian model supports A/B, ABC, and ABCD tests in a unified way. The prior structure and parameterization are consistent across:
- Two‑arm comparisons (A/B).
- Three‑arm comparisons (ABC).
- Four‑arm comparisons (ABCD).
This consistency simplifies both implementation and interpretation. The same prior assumptions about conversion probabilities apply across different numbers of arms, and domain knowledge from HelloFresh about plausible conversion rates was embedded directly into the model. This unified approach, combined with the unpooled multi‑dataset design, means that:
- All tests, from simple A/B to four‑arm ABCD, can be run in one coherent system.
- Sampling remains efficient thanks to the improved model structure.
- Inference remains accurate, as seen in the parameter recovery simulations.
Where Can You Learn More?
A/B testing with Bayesian methods has been widely discussed. For readers interested in background material and related work, examples include:
- A PyData talk on applying Bayesian statistics to A/B testing in business contexts.
- A blog post on Running A/B tests with millions of observations
These resources discuss general Bayesian A/B testing principles and provide additional examples of PyMC‑based workflows.
Conclusion
HelloFresh needed a scalable, trustworthy Bayesian A/B testing pipeline that could handle thousands of concurrent tests efficiently. The original setup suffered from slow, correlated MCMC sampling and high computational costs. By:
- Redesigning the Bayesian model with fewer parameters and revised priors.
- Ensuring the model worked consistently across A/B, ABC, and ABCD designs.
- Validating the approach through parameter recovery simulations.
- Moving from many small, separate PyMC models to one large unpooled model that fits all datasets simultaneously.
the overnight batch runtime was reduced from 5–6 hours to about 5–6 minutes. This 60x speedup enables faster, data‑driven decision making, reduces reliance on expensive compute resources, and prepares HelloFresh’s experimentation pipeline for continued growth.
Find Out More
If you are interested in similar Bayesian modelling and performance optimisation work, PyMC Labs regularly publishes case studies and technical posts on probabilistic programming, scaling Bayesian inference, and real‑world applications.
Reach out to learn how your team can integrate and benefit from this approach.
Sources:
- PyData Talk: https://youtu.be/8huhVCMmS_c?si=nPqj5Cx-_Y1bf_5K
- AB testing: https://en.wikipedia.org/wiki/A/B_testing