The only hard part about testing is that there is no "right" answer because the target is always moving. The things that delight customers go in and out of fashion and all you are doing by testing is trying to catch and ride that wave. û John Wall (www.themshow.com)
One of my first jobs in industry, years ago, was as a production control clerk at the Thermoid Corporation in Trenton, NJ. To control production they had a library of 5 x 8 cards, one for each of the thousands of different automotive rubber products that they manufactured. At the top of each card, a senior analyst had written the minimum on hand stock level, and standard order quantity. On the body of the card, junior analysts posted daily additions and subtractions to the quantity on hand. My job was to look at each card every day to place orders for manufacture of any part whose stock level fell below the minimum.
This activity was, of course, before the days of computers. Today this simple process is done electronically, eliminating not only my old job, but the jobs of the analysts who posted the data, and the senior analyst who controlled the system. I am sure that there was some sort of a fight before management could replace them. At the time all of us working there assumed that what we were doing was state of the art - which it probably was. If it ain't broke, don't fix it. The changes when they were made, of course, made Thermoid and every other manufacturing company tremendously more efficient, bringing prices down and profits up.
The same attitude toward change exists today in the direct marketing industry. I recently saw a quote by a junior analyst in a company doing customer e-mail marketing. "Testing: A process designed to: 1) make marketing more complex, 2) slow down production and 3) prove to that pigheaded boss that yes, we really do know what already works." I even heard a senior marketing VP at another major corporation with 25 million annual customers say about a year ago in regard to their daily direct mail program, "We don't need testing. We know what works."
More people than you would think have the attitudes of the junior analyst and the marketing VP. Yet, without testing, it is almost impossible to improve. What we are doing may work, but would something else work better? That is the question that cannot be answered without constant testing. If we do not test, our competition may be testing, and learning something that we have not discovered. Our perfectly good direct marketing program may be beaten by our competitor's even better one.
Testing can involve considerable internal corporate contention. If your regular promotion produces a response rate of 2 percent and your test produces only 1 percent, you will be accused of wasting the company's money by doing the test. If your test produces 3 percent, you will be accused of wasting money by mailing your regular 2 percent promotion. If you do no testing at all, no one will complain. This situation makes it difficult to get tests approved.
Control group problems
To test in the e-mail and direct mail industries you have to have control groups. Control groups are folks who get what you have always sent, while test groups get something new and different. You learn by comparing the response and conversion rates of the two groups. Each group has to be large enough to give statistically valid results.