Common Multi-variant Testing Mistakes

As promised, I’m getting on with the Trusted Expert series and finishing up with the last few posts before everything moves over to the new site. When we left off, we were talking about testing, and the next post is about multivariate testing.

And I’ve been dreading it – not because multivariate testing isn’t worth doing, it so is, but there’s a reason why I’ve been stuck at this particular point for some time. You see, multivariate testing can be a valuable tool that can tell you some very important things about what is working on your site and what isn’t. No, multivariate testing can be an extremely useful thing -when it’s done right.

But if it’s done wrong, it can be a colossal waste of time that can also lose you money. And unfortunately, it’s easy to get multivariate testing wrong. And since I’m writing this post on multivariate testing, if I give you the wrong advice or don’t explain myself clearly, it’ll be my fault. No pressure.

So this post on multivariate testing will strive to be as clear and concise as possible. While firmly stand behind the importance of testing, I in no way claim to be any kind of testing guru or expert – my aim here is to give you a good grasp of the basics so that you’ll walk away with a good understanding of how things work, how it relates back to being a Trusted Expert. And if you want to go ahead and delve further into the more technical aspects of multivariate testing once we’re done, you’ll have a good knowledge base with which to do so.

So without further ado (that had to be the longest intro to a post, ever) let’s get into the nuts and bolts of multivariate testing. If you remember from way back in my post about A/B testing, in that case you can only compare one thing at a time – it’s built right into the parameters of the test. With multivariate testing, there is no such restrictions, so it’s up to you to make sure the things you choose to test don’t actually automatically negate any results you get.

As the name suggests, with multivariate tests, you’re dealing with more than one variable, and you can have any number of variations on each of those variables (more on whether that’s a good idea in a moment). Remember, with an A/B test, you could have any number of variations, but they all had to be about the same element (headers, checkout button, images, etc.) When you’re running a multivariate test, one would assume that the variables you choose to test together have some sort of relation to each other that will result in an overall improvement in performance, but that’s up to you to make the right choices (see why multivariate testing is so easy to screw up?).

Multivariate tests allow you to test the power of different combinations on your site, which can be a very powerful thing. We all know that all the elements on your site work together to influence the user, but with multivariate testing, you can figure out what combination of items will give you the best result.

And with multivariate tests, you get multiple possible combinations. For example, if you’ve got two areas you’re going to test, (say headlines and button color) and you’ve got 3 variations for each – that means that there are 9 possible combinations (3×3=9) that need to be tested. The more areas and variations, and the combinations will grow exponentially.

If you want to get your feet wet in multivariate testing, a good test to start with is a headline/main image combo, since they are viewed together and definitely have in impact on each other. Stick to no more than 3 variations for each – this will result in a test that will not take too long and give you results that are easy to interpret.

Since multivariate testing allows you so much more freedom than A/B testing, it can be easy to run into problems. Here are some common mistakes that people make:

testing too many things at once. When you start out with multivariate testing, it can be tempting to test a whole whack of things at once – Google Optimizer allows you to choose up to 1,000 possible combinations. The problem with this is trying to many things at once can muddy the waters. If your chosen combination has 5 different elements, it could be that, even amongst your highest performing combination, there are still things that aren’t working for your site. But since you had so many elements, it can be hard to pinpoint which one was the problem.
choosing too many variables. Similar to choosing too many elements, choosing too many variables will give you a lot of combinations – and that will make the test take longer. Some tests, because of the sheer number of combinations, can take YEARS to finish. Do you have that kind of time? Didn’t think so. Keep it simple, and see test results you can actually apply to your site in time for it to actually make a difference.
losing sight of your goals. Remember, going into any kind of test, you want to have some kind of hypothesis – the outcome that you expect. You need to keep that in mind when preparing any multivariate test so you don’t get mired down in the sheer amount of details. So for your headline/main image test, your hypothesis could be that an action-oriented headline along with a dynamic picture will be the combination that works best. You then use the multivariate test to either prove of disprove your hypothesis. And, as always, you apply that learning towards your next test and hypothesis.

Hope that helps you understand multivariate testing a bit. As I mentioned at the beginning, it can be a very useful tool to finding out what will perform the best on your site. But because of the sheer amount of freedom multivariate testing gives you, you need to make sure that you practice some kind of restraint to ensure that you’re getting the best possible results.