If you’ve spent any time working with a marketing professional on any of a number of different areas: direct mail, e-mail newsletters, banner ads, websites, and so forth, you have probably heard either “A/B testing” or “multivariate testing” used by the marketer. They may or may not have explained what this bit of jargon means, however, and we want to make sure that you are properly informed of as many things as possible.
Let’s not kid ourselves: Sometimes, professionals have a bad habit of just assuming people know what all of the jargon in their industry means. It’s important for us to be conscious of this and be ready to educate anybody who asks us what a specific term means. As a result, it’s also important for us to have a very clear idea of what the terms we are using actually mean. Nothing hurts a professional’s credibility more than to be asked what they mean by a certain term, only to find themselves unable to adequately explain the concept to a person outside of their profession.
So, A/B testing and multivariate testing are two terms which mean very similar things. In fact, you could say that A/B testing is a specific type of multivariate testing. In this case, A/B testing is a more specific term than multivariate testing. So, to adequately explain what A/B testing is, you kind of need to start by defining multivariate testing.
Let’s do that right now. Multivariate testing, which is sometimes also called bucket testing (I know, you’ve got to love it when the same thing goes by many names), is basically the act of creating multiple versions of the same content. So, in the case of a webpage, a multivariate test would involve creating numerous layouts of the webpage. The goal here is to display different versions of the same webpage to different (randomly sampled) people, so that you can get data on each of the different webpages to see how well they perform. There is no theoretical limit to the number of webpages you can test in this way; there are only practical constraints, since you must get a sufficiently large sample for each of the tested items in order for the data to be meaningful.
Suppose you wanted to try three homepage layouts for your website. Multivariate testing is the process by which you carry out this test: Essentially, by making all three versions of the layout live, and determining randomly which version someone sees when they navigate to the homepage URL, you can then set analytics up on each homepage and view stats such as hits (important in this case because it shows sample size), bounce rate, views, and so forth. The result is a scientific way of determining the top performer among the layouts you are testing.
So, that’s multivariate, or bucket, testing. What about A/B testing? Well, like I said earlier, A/B testing is a specific type of multivariate testing. You can probably guess what the special case here is, but I’ll go ahead and say it anyway to confirm your suspicions. A/B testing is simply multivariate testing which tests exactly two versions of the content being tested, again, be it e-mail, direct mail, website, or other marketing collateral. A/B testing is particularly useful because of its ease of implementation compared to larger multivariate testing, and is good for comparing two new concepts to replace a decidedly failed old concept. What it’s really good at, though, is demonstrating that a new concept actually outperforms an old concept. If the old website is the “A” in an A/B test, the “B” (or new) website is displayed to approximately half of visitors and the old to the other half. Again, the results are analyzed, and with any luck, demonstrate that the new website concept outperforms the old, making it the better choice moving forward.
In short, multivariate and A/B testing are simply two ways of comparing different versions of the same marketing collateral in order to pick out the best performer. Multivariate tests many different versions, while A/B testing specifically refers to the testing of exactly two versions. We hope that this helps you have a better understanding of your marketer’s use of these terms. If not, let us know! We’re always happy to answer questions about what we do.