Upside Downside of Taguchi Tests Revealed in Progress

Okay, so there are a couple good things and a small number of bad things which I find are happening in the use of Google Website Optimizer for the Taguchi tests.

First: WEBSITE OPTIMIZER NOTE: Your collection code for the goal DOES have to be after the analytics tag, however, it DOES NOT have to be before the 'body' tag.

Pros

Its FREE!

Tests are progressing very quickly. I actually had to shut down the test, copy and restart twice since the original post because it ran into some tag deletion from an update in our site from a saved version. It made for a frustrating couple of days until I could clear up the data. However, once they get out there, they run very quickly with limited traffic exposure and you can begin to see projections, albeit with a significant confindence interval, within a couple hours.

Picture of Element Value is Much Clearer. When we built the test variations, we purposely built in a few 'dogs' which we knew were going to be terrible indicators of conversion to each of the element variations. These include purposely bad spacing, terrible wording, image problems, and excess text. I mean, we really tried to get an idea of how bad we could make something. In the variations where there are clearly bad elements, there is corresponding poor performance. This is wonderful as far as I'm concerned because it validates our hypotheses about conversion driving elements. Paul Holstein actually put out a post to the Yahoo! group on these which is available at: http://tech.groups.yahoo.com/group/webanalytics/message/9383
We're pretty sure the list is worth printing and having prior to testing.

Cons

Editability is much more complicated. With the boons of quick results and dynamic testability, come the burdens of human development. Errors which have not been discovered prior to launching the test create the possibility that the test is not going to give a clear picture. When things DO go wrong, it wipes out whatever progress was made with the test and you have to copy, then restart the test with the corrections. With the single element, full factorial tests, you will be able to make changes to the elements which are not tied into the html directly (say an image needs to be clearer or its link needs adjustment).

Element Strength is Combination Strength. With Taguchi tests, the combination is the element. This means that the test is geared toward optimizing the best possible combination and not the strength of a single varied element While this falls under the CON section, its only a conditional con as it makes for great follow up testing for your traditional uses of the Website Optimizer.