Case Study: Automating Optimization

Screen Shot 2015-09-04 at 10.03.50

(This is a business case study. It will be used to guide discussions during the session: “Automating Optimization” at the Vendo Partner Conference in Barcelona on September 17th.)

automating optimization

Matt had a headache. Silvia from HR offered him an aspirin. He took it, drank it with a glass of water and waited for it to kick in. While he waited he listened. “Yes, I can find the people you are looking for…it’ll take me a few months, but the real challenge is that we’re going to need more space,” Silvia said. He understood the challenges of adding people, that communication and coordination become exponentially more difficult with each new person. And now he was facing another challenge. Space. The next frontier. And he didn’t want to go there.

He thought back on what had brought him to this office-bursting moment. “I had had a relatively open afternoon. I was leaning back in my chair and taking a look at some of our traffic numbers. They were ‘great’ numbers – evidence that we were in the lead, ahead of the game. But were they really telling such a good story? The truth is I didn’t know. What I did know was that we can always do things better, so I was a little skeptical,” recalled Matt.  

Many years ago the industry’s approach to traffic was simple. Every visitor arrived at the home page. “All our traffic went there, whether it came from search, from an affiliate’s site or from ads. That’s just how it was, and ‘how it was’ was working well – so why change it?” Matt recalled. Why, indeed?

He wondered if different traffic sources might indicate something about how the surfers would behave. If that was the case, perhaps he could adjust what he showed surfers based on where the came from to increase conversion in each group. It took him and his team months but they found the answers. “Yes, there were real differences, and, yes, we could optimize conversion by giving each traffic source its own experience,” Matt recalled. His team rotated different product images based on the highest click through for each traffic source. “Manually it would take several man hours on a daily basis. We had lots of sites and lots of traffic sources so we hit the limits of what our staff could do pretty quickly,” Matt remembered. And so he found himself sitting in a conference room with Silvia talking about…office space. And the aspirin still hadn’t kicked in.

Leaving the meeting with Silvia, Matt walked down the hall to the office of his tech lead, Phil. “I explained the situation to Phil. The whole thing from start to finish. He understood the challenge we were facing. Within days we had a solution to display and rotate images automatically based on source and click through. We’d automated a heavy manual effort. It felt great. And I could stop looking for new office space,” recalled Matt. The automation also freed up resources to find the next big improvement: PC versus mobile surfers, 4G versus Wi-Fi connections, surfers in New York versus surfers in Barcelona, etc. He was on to the next rotation of the never ending lifecycle…optimize, automate optimization, and then optimize some more to adapt to a changing world.

As he started to scale up his automated optimizations Matt began facing new questions. “Are ten automations 10x more effective than 1? Is the 1st still just as effective once you get to the 10th?” Matt asked. Experience had taught him that some automations are more effective than others. Certain automations may needed to be combined or merged. Others were not worth doing at all or became deprecated. “The only way to know if I’m making progress is to regularly review and re-assess. As well all know, time (and in our case, technology) changes things,” concluded Matt.  

Matt summarized the questions he’s asking today about automating optimization:

  • How do we identify areas of improvement?
  • When do we begin thinking about automation?
  • How do we handle the seemingly never ending combinations and required resources to manage them all?
  • What tools do we or have we used to organize and execute them all?
  • How often do we go back to look at that first automation to make sure it’s still relevant or productive?
  • Where would we be today without automation of optimizations?

Questions for discussion: How can I code to continually test and optimize? What are different approaches? How much do they cost and how much of a lift will I get from them? In other words, when is the juice worth the squeeze?

Leave a Reply

Your email address will not be published. Required fields are marked *