Running experiments can be a messy business and you often want to make changes mid-experiment.
Making changes to running experiments in GrowthBook 2.7 is:
Safer than ever. With guided flows that ensure you don’t introduce bias when changing targeting or traffic rules, you are able to pick the least disruptive deployment strategy for your changes.
More flexible than ever. Sticky bucketing allows you to preserve user experience when making certain kinds of changes.
This articles walks you through three kinds of changes you can make in GrowthBook, how our UI helps you navigate them, and how sticky bucketing can help you make changes safely.
Decreasing traffic to an experiment
In some cases, you may wish to decrease traffic to an experiment, either because you have enough users and want to ramp down new enrollment, or you want to begin restricting your experiment to some subset of users using more restrictive targeting attributes.
Imagine you add a targeting attribute to only target US-based users for your experiment. If you simply make the change and push it live, then your experiment analysis will include users from before you pushed the change and they are no longer eligible for the experiment. They will still be in your experimental sample which means their behavior may be affecting your experiment results, but they’ll no longer be receiving the same feature values as before since they are now excluded by the new targeting rules.
In this case, you have three options:
New phase, re-randomize users. This ensures your analysis is accurate, but it throws away your existing data and it may change users’ experiences.
Same phase, apply changes to everyone. This lets you leverage existing data with the knowledge that some users may be in your experimental data but are no longer receiving the new experiment feature, therefore biasing your results.
In GrowthBook 2.7, we have added sticky bucketing, which, if enabled for your org, provides you a third option: Same Phase, apply changes to new traffic only. This lets you (a) keep all of your data in your analysis, (b) ensure users do not change their originally assigned variation, and (c) avoid suffering any bias as users in each variation will continue receiving those feature values.
Here’s a loom showing the flow in action
Increasing traffic to an experiment
Increasing traffic to an experiment is less problematic. You can safely make any of the following changes without starting a new phase:
Increasing the percent of traffic to an experiment
Removing restrictive targeting (i.e. making targeting more permissive)
Removing an experiment from a Namespace
Restarting an experiment, or starting a new phase
Sometimes, we need to start an experiment over and throw away our old data, either due to a bug in implementation, a change in experiment design, or if we want to restart an A/A test to make sure that some imbalance was due to chance and not due to an issue with your GrowthBook implementation.
In these cases, starting a new phase of the experiment will require that you re-randomize in order to avoid carry over bias. You can read more about carryover bias in our docs.
As with other changes, we provide this information directly in the app to ensure you can make fully informed choices.
Feel free to check our our docs on sticky bucketing or making changes to running experiments for more detail.