Sitecore Multivariate Testing is a Marketer’s New Best Friend

Testing and optimization are the mantra of any online marketer worth their salt. Good marketers spend a huge amount of time tweaking and testing their page content, images and calls to action to dial in the most effective combinations.

As part of Sitecore DMS, the multivariate testing tool offers a powerful option for marketers to measure and optimize their site’s performance. In today’s post we’ll review some of the features of the tool and compare it to the A/B testing tool available with Google Analytics.

Sitecore Multivariate Testing Overview
Unlike the previous versions of Sitecore marketing tools, the Multivariate testing feature in Sitecore’s DMS is quite robust. You can test multiple versions of the same type of content item against each other to determine which performs best. For instance, you might have five different homepage callouts that you want to test to see which produces the highest click through rate. With Sitecore’s Multivariate testing, you can assign these callouts to the same location on the homepage and Sitecore will automatically rotate them to different users to see which creates the highest click through rate. Alternatively, you also have the ability to test multiple page elements at one time. In other words, you could test three banner images, four different versions of body copy and six different calls to action on a landing page to see which combination of these content components generates the best conversion rate from users.

In addition to providing a wide variety of testing options, Sitecore Multivariate testing is fairly easy to use. Accessing the tool is simple, you can set up new tests and review statistics for active content tests through the testing menu in the Page Editor ribbon or from the Testing Lab, located in the Marketing Center. The tests are also fairly straightforward to set up, an experienced user can launch a basic test in less than ten minutes once the alternate test content items have been created.

Learn how to create a multivariate test and review the results by watching the latest tutorial Multivariate Testing in DMS.

Once set up, Sitecore will track the performance of each of the components you’ve assigned to the test. At any time during the test you can view the performance of each component by going to the testing menu in the Page Editor ribbon or to the test lab in the Marketing Center. When you’re confident you have enough data to make a conclusive decision about your test, you can end the test. Once complete, you can select the best performing components of the test and assign them to the test page permanently. Or, if you’re not satisfied with the results, you can re-start the test from the testing center.

Tool Comparison: Sitecore MV Testing VS. Google Analytics
Prior to their Sitecore implementations, a lot of our clients are most familiar with Google Analytics. A common question we get from advanced analytics users is how the testing features of Sitecore compare with Google Analytics. Here we’ll provide a quick comparison of the two tools comparing the pros and cons of both.

Setting up a Test
Edge: Google Analytics. Because of Google’s basic testing features and simple, straightforward user interface, setting up a test in Google Analytics is simple. All you have to do is use Google’s URL generator to develop unique campaign URLs for each inbound piece of content. While Sitecore is fairly easy to set up, it does take more time to get a test up and running than with Google Analytics.

Testing Options
Edge: Sitecore Multivariate Testing. Sitecore wins hands-down for the available options and flexibility of testing. Google Analytics is limited to basic A/B testing for traffic coming from multiple landing pages. Sitecore can test the performance of similar landing pages, individual content items within a landing page (such as different versions of a callout) or multiple different content items within a page (callouts, body content and images) all at once. Because it isn’t integrated with the content management system, Google Analytics is not able to track the performance of individual page components as accurately as Sitecore. It can be done, but requires you to manually combine the data from multiple, separate tests in order to get an accurate picture of performance.

Monitoring Test Performance
Edge: Sitecore Multivariate Testing. Rather than just providing data on which page generates the most click traffic (as Google Analytics does), Sitecore can tell you which combination of page components on each landing page is most effective at generating clicks. Because it can measure the performance of individual page components, Sitecore is able to provide a more holistic picture of page performance and how individual page components work together to generate the best possible outcome.

Test Completion & Updating Site Content
Edge: Sitecore Multivariate Testing. Perhaps the most persuasive component of Sitecore’s testing platform is the ease of implementing test elements. Because the testing tool is integrated with Sitecore’s CMS, you can immediately implement the winning page version, component or combination of components that performed best in your test.  Even better, this can be done directly from the testing menu without having to navigate back to the content editor interface. In comparison, Google Analytics requires you to make all of the updates to your site manually once the test is complete.

While Sitecore’s Multivariate testing tool requires a little practice to master, it provides a wealth of options that non-CMS integrated tools simply can’t match. If you’re a marketer with Sitecore’s DMS toolset, you will find the testing and test management options available extremely useful and efficient for managing your site’s testing and optimization! If you’re considering implementing multivariate testing as part of your DMS toolset and need some assistance, check out our suite of Sitecore Specialty Services on CEP and DMS planning.

Sitecore Digital Marketing Suite (DMS), Sitecore Experience Marketing, Sitecore Client Engagement Platform (CEP), Sitecore training, video


Add a Comment


Please confirm you are human by typing the text you see in this image:

StephenG said: 5/16/2013 at 8:04 PM

It sounds great! I do have some questions about how tests work in Sitecore.

- Is it possible to see (and adjust) how much traffic is visiting each recipe?
- Can I see the conversion rate for each recipe, or is Sitecore’s calculation of “value” the only way it determines a winner?
- How will I know when the test has reached statistical significance, and will the test stop automatically at that point? Essentially, how can I calculate when to stop a test? This is done automatically by Google Content Experiments.

Rob McChane said: 5/21/2013 at 9:20 AM

Is it possible to see (and adjust) how much traffic is visiting each recipe?

Unfortunately not, Sitecore manages the test by automatically rotating the content variations to determine the item or combination of items that leads to the best result. If a site administrator was able to determine which version of the test content a visitor saw, the test would not provide an accurate representation of the best content in the test.

Can I see the conversion rate for each recipe, or is Sitecore’s calculation of “value” the only way it determines a winner?

Sitecore measures the success of each test based on the value score that is assigned to the desired action. At this time Sitecore does not provide a conversion rate or total number of impressions that each content combination receives. However, because each content combination is displayed equally, the option with the highest score will also inherently have the highest conversion rate.

How will I know when the test has reached statistical significance, and will the test stop automatically at that point? Essentially, how can I calculate when to stop a test? This is done automatically by Google Content Experiments.

The test does not stop automatically, the site administrator must determine what the correct duration is for each test depending on the unique circumstances related to that test. Once the test has run for a sufficient duration, the site administrator must stop the test manually. Generally, we recommend testing a component for several days to a week (depending on the amount of traffic the test content receives).