Why is taguchi method better than randomness? Best answer on the web

Posted in: darrelrussell.com edit
08 Jan 2009
  • I'm having a hard time understanding why the taguchi method for
    determining the best option for a variable works better than randomly
    selecting an option for each variable and testing it.

    Here's a primer on the taguchi method:
    http://www.offermatica.com/media/articles/landing_page_optimization_using_the_taguchi_method.html
    Here a link to a spreadsheet which shows how to make the "recipes":
    http://www.kowalick.com/freetaguchi/download.html

    After reading those, it seems to me like this method has no advantages
    over randomly selecting an option for each variable each time the page
    loads. Here's my argument:


    Pretend you have 3 variables with 4 options each.
    Total number of unique combinations is 4x4x4 = 64.

    Testing all unique combinations requires 64 tests and gives 16 tests
    per any given option.

    Consider randomness, ie: on each load of the page, there is a random
    selection of each of the 4 options for each 3 variables. If we run it
    64 times then each variable would have been shown 1/4...0.25 x 64 = 16
    times, so there are 16 tests per any given option.

    Consider taguchi: Let's say it creates 16 recipes on the data.
    For a visual, each column is a recipe:
    Var A 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4
    Var B 1 1 2 2 3 3 4 4 1 1 2 2 3 3 4 4
    Var C 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
    If we run 64 tests, then each recipe is randomly selected probably 4
    times. Assuming each option was used 4 times (equal share) then there
    were probably 16 tests per any given option.


    So, the results look the same: all of them had a total of 64 tests and
    all options have 16 tests each. However, I feel the first 2 are better
    than taguchi because every option got a chance with every single other
    option. Look at the recipe - the combination (1,4,4) will NEVER be
    tested, where with randomness and all, it will be tested.

    So, please confirm I am right or explain why I am wrong. Thank you.



  • I still argue that Taguchi only is useful in situations where there is
    a cost involved in making the permutations - that's why it's so
    popular in manufacturing - but when things can be dynamically rotated
    via a webpage at no expense, there is no advantage (in fact there is a
    loss of integrity since the combination of variables is limited to
    specific variables in their recipe).


    I agree with you - subject to one small proviso
    - eliminating ridiculous combinations

    I also suspect that the Taguchi method is what I call a 'benign con'
    - a means of preventing research being a slow meticulous dragnet, rather than testing the 'envelope' - in other words a justification for trying out 'hunches' and not allotting blame if they fail.


  • Thanks for all your input frde, you've been very helpful.


  • I'm starting to think that the only situation taguchi would be best is if it costs resources to make a recipe. For example - if you are testing out various posters, it costs money to redesign different ones, print them, etc. However, when the media is a website and it doesn't cost anything to randomly rotate content, then I don't see the advantage


  • Here's my response to my friend's argument:

    >Let's say then that you need a group of 100 visitors to give it a solid test. That is 100 x 64 = 6400 hits needed. You're working it the wrong way - if you get 100 visitors, and each variable has a 1 in 4 chance of coming up, then you'd get 100/4 = 25 tests for each variable.
    >Vs Taguchi which gives you 3 recipes x 100 visitors to give solid test. That is 100 x 3 = 300 hits needed. First off, you can't have only 3 recipes and be able to get any data for all variables - 1 would be left out. You need a minimum of 4 recipes. Example given: Var A 1 2 3 4
    Var B 2 3 4 1
    Var C 3 4 1 2
    So now if we have 100 visitors and each variable has a 1.4 chance of coming up, there are 100 / 4 = 25 tests for each variable. Where's the advantage?
    I still argue that Taguchi only is useful in situations where there is a cost involved in making the permutations - that's why it's so popular in manufacturing - but when things can be dynamically rotated via a webpage at no expense, there is no advantage (in fact there is a loss of integrity since the combination of variables is limited to specific variables in their recipe).


  • You don't need 64 separate sites
    - just 64 HTML pages
    - and even then the pages could be 'assembled on the fly'
    - each visitor gets a different page

    The 'cost' would be the setup (programming)

    The other 'cost' would be lost revenue from putting up unattractive pages.

    Personally my opinion is that the 'success' of a web page is largely down to its content - and that 'over designed' pages simply confuse people. But maybe I'm biased - I dislike Flash, animations and Java, although Style Sheets have their moments.
    Good luck with your project.


  • Interesting

    I skimmed the link you gave on Taguchi and got the impression that it is for use when you have significantly more options than you have 'lives'.
    Obviously exhaustive testing is the optimum method
    - and given a large number of 'lives' random selection will tend towards exhaustive testing
    If you don't have many 'lives', then with random selection of tests one risks just testing a 'related cluster'
    I have heard of something similar from neural networks called 'heating', where you radically change all the variables - big time. And see what happens.
    My impression is that the general idea is to 'home in' on results that are unexpected, and then examine (further test) in more detail. ie: it is not meant to be used mechanically - rather the results require intelligent evaluation.
    It makes a lot of sense as a means of 'using limited resources'


  • Here's a comment from a friend of mine:

    "The trick is you need to see a number of times for each variable group to run. So, where on the 4x4x4= 64 that means you have 64 seperate sites to run.
    Let's say then that you need a group of 100 visitors to give it a solid test. That is 100 x 64 = 6400 hits needed.
    Vs Taguchi which gives you 3 recipes x 100 visitors to give solid test. That is 100 x 3 = 300 hits needed.
    Each hit costs $0.25

    That measures the difference between the two in THOUSANDS of dollars."