We don’t want to test


We have a reputation for testing. But the truth is we’d rather not test at all. We’d prefer to just know important things. Things like the best price for each consumer. But we don’t. We’re ignorant about lots of important things. So we test. But we test as little as possible.


Testing costs you money

A simple example. Testing three variations of a price ($25, $30 and $35, represented by the three lines above) for a certain consumer profile means – logically – that we expect that two of the prices are not the best. But which ones? During the test we send consumers to prices that will make us less money. Not good. Very expensive. So we try to stop testing as soon as we can.


Ignorance costs you money

We see testing as the lesser of two evils. Test or remain ignorant? Either we test and are certainly wrong for a short period (by definition two of the prices are worse than the other price) or we live on in ignorance (where we are definitely wrong about the price for millions of consumers). While testing is a pain in the ass and expensive, ignorance is quantifiably worse. Test results tell us how much worse it is.

Stop testing ASAP

So we try to test as little as possible. The bare minimum. We want the right answer as quickly and cheaply as possible. In the pricing example the right answer means that we are 90% sure that we are giving that consumer the right price for him. When we’ve got it we stop that test ASAP.

How do we know when we can stop testing?

We stop when we are really confident we know the truth. First, we have to understand the results of the test: Are they random? Do they show one price outperforming the others? Would it happen again the same way if we did the test again? We keep going until the results are accurate and repeatable. Then we stop.

To continue with the same test after that point can be as ignorant and expensive as not testing at all.