As an engineer, I like to build things, but as a practical business matter, it’s usually less straight forward. As much as it pains me to say it, these days for A/B testing platforms it is usually more prudent to use a 3rd party platform. In this article we’ll discuss pros and cons of building vs buying A/B testing platforms.
Building
Building your own A/B testing platform has a few undeniable advantages. If you have an engineering team that has the ability to take this on, the tight coupling to your code and deployment systems can create a great developer experience. Tests can be deeply integrated into the code, unlocking some really interesting and complex experiments. This can bring the A/B testing deep into your product process, and can drastically speed up testing time. Coupled with a feature flag or similar deployment system, you can test almost every feature and update very easily. If you have an existing data warehouse, and if you’re considering building your own A/B testing platform you almost certainly do, you can tie your system into these existing metrics. If you have an aggregate metric or custom leading indicator of value to your users, you can fairly easily test against this directly. High performance sites typically leverage a lot of caching, which can cause problems when A/B testing. A custom built solution can account for many levels of caching without affecting performance too much.
On the other hand, building has its disadvantages. It will take some time to build a good, working system. At my previous workplace, we were continually improving our system. We managed to get a simple system working in about a month, but this had a terrible or nonexistent UI, and many bugs and errors. Some of these bugs took months to find too, resulting in us having to throw out many tests. Like most software, we were never finished, but I’d estimate the stable, production-ready system took about 3 to 5 developer months.
A/B testing is full of counterintuitive results, and if there is any doubt that the platform could have messed up the data or implementation, you can bet it will be blamed. Building trust is vital for a successful A/B testing program, so this is a major problem with homebuilt systems. You may have to contract with statisticians to make sure the math is correct, unless you have expertise in your data team.
Building a front end testing system is quite hard to do, so most internally developed platforms will opt for only back end implementations. This isn’t usually a problem except when inserting engineering as the bottleneck for launching tests — even simple ones.
Buying
Buying comes with some very obvious positives. Typically you can be up and testing in a matter of hours from signing the contract. The statistics engines are proven, which means leadership is more likely to trust the results are accurate. The ease of use of the platform can let a greater range of people participate in your testing program. Front end tests can be implemented without any engineering help.
However, there are also many problems with existing 3rd party platforms. Usually they use their own event tracking, meaning that it’s impossible to test against your existing metrics or data warehouse (Growth Book, by the way, does let you do this). It also means yet another place you’re sending your user data, which might lead to you being out of compliance with the various privacy laws. The kinds of metrics you can test against are typically limited too, with most only supporting binomial events (yes/no, or clicked/didn’t click).
If you open up the platform for many of your team to start tests, you can end up with interfering tests and make the results meaningless. You also run the risk of running it like the wild west, where the tests make no sense, are out of your design style guide, or lack strategic direction.
Front end testing can cause pages to flicker as they load as the payload is injected into the page. Flickering text or buttons can cause the results to be meaningless, as it either slows the page down, or causes users to notice flashing parts more. Deeper code integration for back end testing is harder to implement as it can be a round peg into a square hole. Often the back and front end analytics are not talking to each other.
The biggest impediment to buying, as with most things, is the cost. Many A/B testing platforms charge in a way that increases the more you test. Since you want to be testing a lot, this can mean that things get very expensive. Other solutions, like Google Optimize, are free as long as you stay under their usage limits, but once you go over the limit, then it’s $150k. It is also quite limited in terms of the tests you can run. It’s not uncommon for a medium-sized site to have the cost of the A/B testing platform be equivalent to one quarter to half of a full-time developer’s salary.
Conclusion
To build vs buy can be a hard decision. A/B testing platforms have come a long way over the recent years, and are adding enough features that you should think really hard about building yourself. The costs of building and maintenance are probably much higher than you expect.
By the way, we built Growth Book based on our experiences with exactly this decision — and it addresses all the negatives addressed above. We tie into your existing data infrastructure, are built to make a seamless developer experience, have a front-end test editor, and have tools to encourage a culture of experimentation. We have years of experience doing experimentation and building these tools. We offer all the advantages of building your own system, without the negatives of buying. Spend your time on products that add value for your users. Contact us and let us show how we do A/B testing better.
