The Importance of Experimentation

Research and Test

What my vacation taught me about being a better marketer

Summer is upon us, and with that, summer vacation! I love vacations with the family. My two sons are becoming good travelers. And as such, my wife and I felt it was time to start traveling to more “exotic” locations rather than visiting Disneyland or Hawaii. So, we tried our first travel experiment last year by traveling to Rome, Italy. We told the boys that this trip would be more about the history, the culture, and the food of Italy. They understood that the activities on this trip would be different and accepted the forced broadening of their experiences. By all accounts, our Rome trip was very successful. The boys soaked in the culture, the food, and the history.

This year’s travel experiment was Paris, France. The trip was slightly longer than last year’s vacation, but with similar goals in mind. The outcome, however, was not as picture perfect. The kids complained about the long lines and the excessive walking. And this caused the kids to fight more with one another. The weather was also not cooperative, as it rained several days during our visit. On one day, we were caught in a torrential downpour, getting soaked to the bone. This only added to our kids’ displeasure. And to top it off, my wife got separated from us in the Paris Metro without a phone. We proceeded to spend the next several hours trying to track her down (we eventually found her).

So, two travel experiments and two different outcomes. As I was returning from Paris, I reflected on both vacations. It got me thinking about the importance of experimentation and how running experiments leads to new insight. Experiments are great because they allow us to learn about what works and what doesn’t. And even if something fails, there is tremendous learning in failure.

Experimentation and your casino

This is precisely why experimentation is so vitally important. Every engagement point with a casino customer gives you an opportunity to learn. The engagement points and the value created can be measured and analyzed. And as you run multiple experiments, you incrementally learn more about what works and what doesn’t. For example, you could run experiments on the types of promotion offers that resonate with your player database; you could create a food promotion and test how it performs against a control group. Or you could test how different promotion offers perform against one another.

Creating a culture around experimentation has numerous benefits for a marketer. First, in an experimentation culture you kill off the HiPPO – an acronym for the Highest Paid Person’s Opinion. Opinions and experience are fine, but testing is the best way to get to the bottom of a decision without relying on anyone’s gut instinct. If a senior executive has an idea in a meeting, the response should simply be, “Let’s test it.”

Second, more of your marketer’s ideas will see daylight in the form of tests. This is important because you really don’t know what will work until you try something. When it’s easy to try new ideas, your team can stop speaking in the abstract and instead speak about results and next steps.

And finally, your team will become more empowered to take on new ideas without the fear associated with failure.

Some things to consider when running experiments:

  • Start every experiment with a hypothesis and goal (segment x prefers food offers and we believe 20% of this audience will redeem).
  • Evaluate the results of your experiments using a common framework that allows for an “apples to apples” comparison – use the same metrics when comparing results.
  • Run multiple experiments in parallel with small samples – ideally, only alter one variable at a time.

The key outcome with experimentation is learning. Whether the results are positive or negative is inconsequential in some ways. A negative result can and often is more important than a positive result. The negative result is telling you that your assumptions on customer behavior may be wrong. And that insight can help you build a better experience for your player.

I learned a tremendous amount from our France vacation, despite the challenges. I now have better insight for our next vacation:

  • Shorten travel time between attractions
  • Buy tickets ahead of time to limit waiting in line
  • Have an emergency plan in place if any of us get separated

Ultimately, despite the challenges, it was a good trip. C’est la vie.

Az Husain