CUPED's sting: More power, more underpowered A/B tests | Conductrics

The benefit of using Regression Adjustment (CUPED) for A/B Testing is that, in certain cases, it can increase the precision of the experiment’s results. However, if one isn’t intentional in the design and setup, this more powerful approach can easily lead to underpowered experiments.

What is CUPED? Clarifying the background noise in A/B testing

CUPED/Regression adjustment is a way of using pre-experiment data to explain away a bunch of the background variability that would otherwise just be background noise in an experiment, weakening or reducing the resolution of the experiment (how big an effect the experiment is calibrated to discern).

Think of the experiment as like the lab microscopes you used in high school or middle school to look at cell structures (cell walls, membrane organelles, etc.). Using CUPED is kind of like purifying the solution that contains the cells before you look for any structure. If you are able to have a cleaner solution, it makes it easier, at the same power of your microscope, to see the cell structures. If you can’t clarify the solution containing the cells, then because of all the impurities, you won’t have the same visual resolution – you won’t be able to identify the same structures given the same power of the microscope.

The problem is that, as with the prep and solution purification process, if you mismanage it, you distort what you see and the resulting conclusions. Also, for CUPED, you need to have the pre-experiment data. Often, you won’t have anything (new visitors, unknown visitors, etc.), or you have existing visitors, but the goal/endpoint isn’t something that you have pre-existing data about.

When is it best to use CUPED/regression adjustment?

It works best for logged/known visitors with a history, and for events (goals) that are frequent and recurring (like minutes spent watching videos per week or the number of shows watched, etc.). These are things that you have a history of behavior about for each customer, and you can use that to predict for each visitor in the experiment what their expected watch amount would be during the experiment if they were exposed to the control (so the same experience as they had previously). You can then basically subtract the guess from what they actually did during the experiment, and then use that remainder for each person as their score/goal amount for the experiment. That makes everything ‘smaller’ and reduces the variance.

How regression adjustment reduces variance

While the microscope analogy explains the concept visually, the mathematical mechanics rely on recycling your historical data. If you have access to pre-experiment data that is correlated (explains some of the variability) with the upcoming test’s KPI (end-point), then by including this data when analyzing the post-test data, you can reduce the amount of noise, or variance in the final result. This variance reduction can be used to either increase the precision of the estimated treatment effect or reduce the required sample size compared to the unadjusted approach.

The amount of variance reduction is directly related to the reduction in the required sample size for given Type 1 (α), Type 2 (β), and minimum detectable effect (MDE). The amount of reduction relative to the unadjusted approach is based on the following relationship:

$$Var(Y_{ra}) = Var(Y) \times (1 – Cor(Y, Covariate)^2)$$

In this equation, Y is the test’s KPI, and the covariate is the pre-treatment data. For example, if the correlation of the covariate and the KPI is 0.9, then the effective variance of Y for the experiment will be 1 – (0.9)2, or 19% of the unadjusted variance of Y. Since the variance of our KPI drives our sample size, this also means we need only 19% of the sample size that would be required for the unadjusted test given the same Type 1, Type 2, and MDE sizes.

The mathematical catch: Misestimating correlation

Wow, an 81% reduction! Huge if true! But is it true? Well, yes and no. For sure, it is a true relationship between the variance of Y, the covariance of Y and some covariate(s), and the sample size. However, like most things in analytics and data science, the issue isn’t really about the method but in its application and context.

For standard (Pearson-Neyman) A/B Tests, calculating the required sample size requires the following inputs:

• The rate of Type 1 control (this is the α, often set to 0.05).
• The rate of Type 2 error control (this is related to the power of the test, often set to 80%).
• The minimum detectable effect (MDE).
• The baseline variance of the test metric (for binary conversion, this is implied by the baseline conversion rate).

So we can think of basic sample size calculation as a simple function, f(α, power, MDE, variance(Y)). We get to pick whatever α, power, and MDE we want to configure the test, but we need to estimate the variance of Y. If we underestimate the ‘true’ variance of Y, then our test will have less power than what we specified.

For the CUPED/Regression adjustment approach, we need to add an additional estimate for the correlation between the covariate and Y. Our sample size function becomes f'(α, power, MDE, variance(Y), correlation(Covariate, Y)).

Notice that the effect of the correlation is quadratic. This means that any misestimation of the correlation when it is high will have a large effect on the sample size. But this is exactly when CUPED/Regression adjustment is most valuable.

How to tell if a test is underpowered

To illustrate, if we estimate the variance to be 100, but really the baseline variance is 110. Holding our MDE and α fixed, underestimating the baseline variance leads to a slightly underpowered test – instead of a power of 80%, our test would have a power of 76%. So less powerful, but not drastically less.

Let’s say we have some pre-test covariate data and we overestimate its correlation with Y. Let’s use as an example an estimated 0.9 correlation coefficient, since this has been used by others who promote a more indiscriminate use of CUPED. However, rather than having a 0.9 correlation, our KPI during the testing periods is closer to having a 0.85 correlation.

This results in our A/B Test having too few samples, or conversely, being underpowered since we should run the test at (1 – 0.852), or for 27.8% of the unadjusted sample size, rather than only 19% as suggested by a correlation of 0.9.

Simulating the impact of CUPED on test power

Below is the output of t-scores from simulations of 5,000 A/B tests where the actual treatment effect of the test is set exactly equal to the MDE. A test configured to have 80% power should fail to reject the null only 20% of the time. In the simulations, we use the sample size suggested by a correlation coefficient of 0.9. In the simulation we ran:

1. A standard unadjusted difference in means t-test.
2. A regression-adjusted t-test with a covariate that has only a 0.85 correlation with Y.
3. A regression-adjusted test with a correct estimate of 0.9 correlation of the covariate and Y.

Both versions 1 and 2 of the tests are undersampled for a given desired power of 80%.

The unadjusted A/B tests (Purple) are shifted to the left of the 1.645 critical value, with 70% of all tests failing to reject the null for the one-tailed 95% confidence test. This makes sense because we only have 19% of the required sample size. The regression-adjusted test with the correct estimate of 0.9 (Green) has just 19.8% of the tests to the left of the 1.645 critical value, which is the expected 20% fail to reject rate (80% power).

The regression-adjusted tests using a covariate that has only 0.85 correlation rather than 0.9 (Red) are shifted to the left, such that we fail to reject 34% of the tests, for a power of 66%. So, exactly when CUPED/Regression adjustment is most useful, if we are not careful in our thinking and application, it is also exactly when we are most likely to increase our chances of running highly underpowered tests.

The business risk of non-stationary data

Why does this happen? Our estimates of correlation are based on sampling the data we have. Often, A/B tests have different eligibility rules, so each test, or family of tests, will need to have its own correlation estimate. Each estimate requires a faithful recreation of these eligibility rules to filter on the existing historical data.

As we generate many of these estimates, each on a subset of historical data, it becomes more likely that we have a nontrivial share of them with underestimates of correlation. This increases the odds of mis-estimation even if the underlying data is stationary. If the data is not stationary, we are even more likely to run underpowered tests.

Of course, this doesn’t mean you should never use regression adjustment/CUPED. For example, here, a simple, admittedly ad hoc fix might be to use a slightly more conservative estimate when the correlation estimate is very high, especially when it’s based on limited historical data for finer subsets of the population.

However, it does suggest that promises of more advanced methodologies are often broken in the details, and that one should be intentional and thoughtful when designing and analyzing experiments. Good analysts and good experimentation programs don’t just follow the flavor of the day. Instead, they are mindful of trade-offs, both statistical and human factors that can affect the outcomes.

Category: Experimentation