Multi-Variate Testing Info

There are many different methods by which an analytics practicioner can go about making leaps and bounds toward page and site optimization. These include functions at the mindset level of the analyst performing, winning the trust of the design team, and adhering to the principles of science which define our objectives. An analyst must be a business scientist before he or she can call themselves an analyst.

Working with Paul Holstein, I've realized there are a certain number of things which come in very handy. First, test before you speak. Paul is a man focused on results and high levels of confidence. He does not like to make business decisions based on opinion. So, when you sit down to discuss some solid insights, you'd better know damn well how those insights are valuable, where to apply them, how to measure the results, and what to expect in terms of success. Its my impression that Paul is of the mind that science won't let you down as frequently as opinion will. I can't argue with that. It kind of reminds me of math. No matter how you cut it, math is a subject where there is no room for debate. You are either right or wrong. There's no gray area, I like that.

To understand Multi-Variate testing in theory is not difficult. First there are a few functions and properties of the testing which must be uncovered and dealt with prior to making any changes to the site which you are planning on testing. There are a few things you should know.

The first thing to decide is what to test. My suggestion is to create a heirarchy of the pages you feel are most in need of testing outside of any analysis. The purpose of this is to learn the necessity of detachment which must exist. You can list all the pages that really get under your skin, that you hate to look at and which you feel are just hiddeously bad pages for your conversion, whatever that might be. (In my case, I've created an algorithmic formula which can list my best pages from top to bottom taking all my most important KPIs into account with a single sort) After you've done this, run some reports to determine which pages have the highest bounce rates, the most traffic and the least importance within the path of conversion. Sort them in descending order by highest traffic. Within the first ten or fifteen pages, you'll see where you need to start making changes. Keep in mind, if this is your first attempt at the test, you will need to avoid making changes to function pages and pages central to the operation of the site on the whole (i.e. Home Page, Newsletter Forms, Shopping Cart or Checkout, or even Internal Search pages) as this may actually cost you a great deal more by having a single adversely performing test combination.

Compare the list you made before doing a traffic-conversion analysis. Your results may or may not be very interesting, but if they are different, it can be a real eye-opening experience. Consistently, when I produce the list to the boss and the design team, I get interesting reactions. Often they confuse aesthetics for functionality and vice versa. This HAS to be a very common problem as it would be with any internal sections devoted to one hemisphere of the human brain.

After you've chosen a page to examine and test, you should keep in mind that there are some things you can change and some things you can't. In another blog, or possibly the Yahoo! web analytics forum, Paul produced a list of things which will have a greater impact on conversion. When preparing for an multi-variate test, its probably a good idea to print it out and have it close by to draw out how you will attack the methodology. Since this is my first official post, I think I'll cut it here and hope that the response show's enough demand for me to continue with this blog.