Q: “I’d like to know how much more money Vendo makes me than another biller. Can I just compare periods of time (ex. September vs. October)? Basically, I turn you on for one month, then turn on another guy for another month then I compare results. That should work right?”
A: Unfortunately, your answer is “no”.
Why not? It doesn’t give you a clear answer. And you want to test because you want to know the real difference between Vendo and another biller. It’s important to your business to know who makes you more money. Then the decision of where to send the traffic is easy.
Even if your traffic is steady (unlikely) you just have too many other variables other than the biller. What kinds of variations in traffic mix affect conversion?
Industry conversion rate differences of major customer segments
(Source: Vendo industry data, Results for 30 days leading up to November 20, 2014)
- Los Angeles vs. San Jose 2x
- California vs. Oregon 3x
- US vs. India 8x
- Safari vs. Chrome 2x
- 02:00 vs. 14:00 2x
- Returning visitors vs. New visitors 2x
- Mac vs. Windows 4x
Check your stats or Google Analytics and see if you see conversion rate differences are in line with the industry. Also, look into your traffic source data. You could see something like Type in vs. Tube +10x.
Even when your total traffic is steady, month to month, small shifts in traffic mix among these segments have a major impact on conversion rates. Btw, in each case above the first category converts better than the second (ex. California converts better than Oregon).
If you don’t randomize all the variables you have a problem. Those variables can cause real differences and you can’t tell if it is the biller or some other variable.
The only way to know the effect of a biller on your revenue is to isolate the biller and keep everything else the same through randomization (A/B split testing).
And since you are thinking of testing you can check out testing.vendostore.com