Nine Lessons for Better Results
I am sure we can all agree that testing is a great thing, and many of you have probably tried it to different degrees of success. But did you get the results you had hoped for? After decades of marketing and running many A/B Tests in database marketing, I have put together some of the lessons I have learned about creating a culture of testing in Marketing.
1. Get buy-in from management.
Changing from “gut feel” to an analytical approach and creating the culture of testing is a change. Getting the support from management and having that communicated from the top down can make the difference in success and getting the help needed.
2. Involve more people once you have the management buy-in.
You will need to involve other key people for support. Get supporters from other departments and leaders within the marketing department supporting you. This will be the team you can turn to for brainstorming ideas and support on the execution of the tests. I have found getting influencers from the Players Club, Slots, Finance, etc., can make the difference in the success of the tests.
3. Define your testing strategy.
This is a biggie! A common problem is the lack of sound strategy behind testing. We test solutions to problems, not ideas.
- State your goal. What am I trying to accomplish? What is the objective of this test? For example – Increase trips for a local, low frequency segment
- Identify the problem – The Why. Why are guests only coming two trips on average when they are local guests?
- Form your hypothesis
- Guests are going to other local casinos
- Our offer cadence doesn’t entice more trips
- Identify a testable solution – The How. Brainstorm ideas that could solve that problem. For example, the solution to our problem A is that we need to teach our guests about the benefits of consolidating their play to our casino to earn more rewards through the players club program.
- Tactical implementation – The What. Once you have decided on the solution, you can focus on what you will do to make that solution come to life. This is the execution process of the test. Again, for problem A – do you change your message to showcase benefits? Do you create an offer related to a club benefit?
4. Define your success metrics.
It is essential to define your success metrics and hypothesis before running your test. This keeps you from adding your bias into the results. What one or two metrics will decide that this test is a success? Make sure that your success metrics align with your goal.
Again, for the example problem. Trips is probably the critical success metric, but you will want to look at spend also to make sure that the spend levels did not decrease due to the additional trip(s).
5. Get enough conversions for statistical significance.
Make sure that you are getting enough guests into the Test segment and the Control segment to have statistical significance. My rule of thumb is to have at least 100 conversions/trips per test segment. That means 100 trips in the Test segment AND 100 trips in the Control segment.
Understanding the frequency of the group helps here. A low frequency segment will need a larger count of guests to create 100 conversions per test segment.
6. Segment your database accurately for testing.
Segments are important. Different guest types react differently to different versions of tests. Segments should have like guests. Look at Recency, Frequency, Locality, and Spend. Additionally, make sure that you get the Test and Control segments evenly balanced for accurate results.
A random distribution in a segment can still create unbalanced Test and Control groups. Double check the average spend in each group to make sure that the difference is under 1% for accurate results.
7. No such thing as an unsuccessful test.
You will learn from every test. If the test comes back inconclusive, it is still valuable. The test element may not be relevant to your guests, or you need to retest with a more significant or different segmentation.
8. Beware of the hidden variables that affect your testing.
- Do not run simultaneous tests that target the same segment.
- Look at any outliers that may have occurred. A jackpot winner within the Test or Control group will impact the results.
- Unbalanced Test and Control groups will create invalid results.
9. Communicate your results.
As you are performing more tests and learning how your guests react to change, communicate your results. Put the results in perspective for the next 12 months. For example, a 4% lift in trips in this segment was a $20K lift in Net Revenue for that test.
Half of the segment got the test, and had that test been sent to everyone, it would have been a $40K lift. If this test is rolled out, the next year has a potential $480,000 lift in Net Revenue. This is big – Marketing your results is key to success.
Good luck with your testing, and if you would like any assistance getting this up and running at your property, reach out to us for suggestions. Raving specializes in using data to help you make strategic marketing decisions.
