With this release, we focused on user experience and productivity improvements. There’s a completely revamped Dev Tools browser extension, a Power Calculator, an experiment Decision Framework, and tons of UI and SDK improvements. Read on for more details.
Dev Tools Browser Extension
Our Chrome Dev Tools extension has been an essential tool for debugging feature flags and experiments since 2022, but we knew it could do even more. That’s why we completely rebuilt it from the ground up, making it faster, more powerful, and a lot more pleasant to use. There are too many improvements to list them all, but here are some highlights:
- Firefox support!
- Complete design overhaul (including dark mode!)
- Event logs and SDK health checks
- Sync Archetypes from GrowthBook to quickly simulate different users
- New popup entrypoint (just click the GrowthBook extension icon)
Download for Chrome or Download for Firefox to get started or check out the docs to learn more.
Power Calculator
Many existing A/B test calculators estimate required sample sizes for statistical power but lack access to your historical data and specific metric definitions, limiting their accuracy. Recognizing this gap, we've developed our new built-in Power Calculator to provide more precise and tailored estimates.
This tool operates in two straightforward steps:
- Define Your Audience: Specify which users will be exposed to your experiment by selecting a past similar experiment, choosing a segment, or building an audience from a fact table.
- Set Your Goal Metrics: Identify the metrics that matter most to your experiment's success.
Once these steps are completed, the Power Calculator performs quick queries against your data warehouse, providing you with the estimated run time and minimum detectable effect (MDE) based on your historical data.
You can access the Power Calculator from the top of the Experiments page. Leveraging historical data requires a Pro or Enterprise license; however, a manual version is available for free accounts. For detailed instructions and a deeper statistical understanding, please refer to our documentation.
Experiment Decision Framework
It can be hard to know when to stop an experiment and make a decision. To help with this, we’re launching the first version of our Experiment Decision Framework. When enabled, running experiments will now show some additional info:
- Unhealthy - if there are data quality issues or the test is too low-powered
- No data - if the test has been running for 24 hours and there’s no data yet
- ~X days left - how long until the test reaches the desired statistical power
- Ship now - if all goal metrics are positive and statistically significant
- Roll back now - if all goal metrics are negative and statistically significant
- Ready for review - if the test has run for long enough, but there is no clear winner
For now, you must manually enable the Decision Framework under General Settings to get these new statuses. This feature is available to all Pro and Enterprise customers. We are continuing to improve this feature and are looking for feedback, so let us know your thoughts!
Read the docs for more info.
SDK Updates
Some of our SDKs have fallen a little behind, so we’ve been working hard to bring them all up to spec.
Pre-requisite Features and Sticky Bucketing are now supported in the latest versions of all of the following SDKs:
- JavaScript (including Node, React, and Edge)
- Swift (iOS)
- Kotlin (Android)
- Flutter
- Java
- C#
- Python
- Ruby
- Go (sticky bucketing coming soon)
Open Feature is now supported for Web (Javascript, React), Node.js, Java, and Python.
We’ve also released dozens of bug fixes, performance improvements, enhanced thread safety, and more across many of our SDKs. Check out the individual release notes for more info.
We’re continuing to invest more time in all of our SDKs going forward to make sure every language and framework has a top-notch developer experience. If you find something that can be improved, please let us know!
Design System Updates
Last but not least, our design system migration is moving along quickly. You should notice big improvements to dark mode, feature and experiment pages, and more consistency and UI polish in general across the entire app. Let us know if you have any feedback!
To explore all the changes in this release, please visit our release notes.
