Changing Running Experiments Safely and Flexibly in GrowthBook

· 3 min read
Changing Running Experiments Safely and Flexibly in GrowthBook
Photo by Ross Findon / Unsplash

Running experiments can be a messy business and you often want to make changes mid-experiment.

Making changes to running experiments in GrowthBook 2.7 is:

This articles walks you through three kinds of changes you can make in GrowthBook, how our UI helps you navigate them, and how sticky bucketing can help you make changes safely.

Decreasing traffic to an experiment

In some cases, you may wish to decrease traffic to an experiment, either because you have enough users and want to ramp down new enrollment, or you want to begin restricting your experiment to some subset of users using more restrictive targeting attributes.

Imagine you add a targeting attribute to only target US-based users for your experiment. If you simply make the change and push it live, then your experiment analysis will include users from before you pushed the change and they are no longer eligible for the experiment. They will still be in your experimental sample which means their behavior may be affecting your experiment results, but they’ll no longer be receiving the same feature values as before since they are now excluded by the new targeting rules.

In this case, you have three options:

Here’s a loom showing the flow in action

Increasing traffic to an experiment

Increasing traffic to an experiment is less problematic. You can safely make any of the following changes without starting a new phase:

Restarting an experiment, or starting a new phase

Sometimes, we need to start an experiment over and throw away our old data, either due to a bug in implementation, a change in experiment design, or if we want to restart an A/A test to make sure that some imbalance was due to chance and not due to an issue with your GrowthBook implementation.

In these cases, starting a new phase of the experiment will require that you re-randomize in order to avoid carry over bias. You can read more about carryover bias in our docs.

As with other changes, we provide this information directly in the app to ensure you can make fully informed choices.

Feel free to check our our docs on sticky bucketing or making changes to running experiments for more detail.

Read next

Want to give GrowthBook a try?

In under two minutes, GrowthBook can be set up and ready for feature flagging and A/B testing, whether you use our cloud or self-host.