Do You F-Up Your Direct Mail A/B Tests?

A/B testing…

Everyone tells you all the time to do A/B testing.

Without it, you never learn, never grow, never win.

How else will you uncover what really works best: a better price, a better headline, a better image?

So… you test.

But, over the years, I discovered that so many Direct Mailers F-up their tests.

Why?

Because they don’t follow these simple 5 ½ Rules of Direct Mail A/B Testing.

1. Don’t Hide the CTA.

Every marketing guru will remind you to include a great CTA. But too many Direct Mailers hide the CTA because they are either too clever or too ashamed to actually ask for the order!

If you were around in the 1980’s (like I was) you know that every call to action started with: “Here’s what you should do now” followed by numbered instructions.

The cleverest we got was, “Here’s how to get your widget in the next 7 days:”

Then came the 1 – 2 – 3 of the CTA.

1. Fill out the form below.

2. Include your payment (make your check out to Acme Anvils.)

3. Use the handy reply envelope enclosed and rush your order to us.

Simple, clear, step-by-step instructions.

I even increased orders in a test – that later became the control – by adding a new number 1 instruction: “Take out your pen.”

Yes… that simple, and very obvious step increased response.

Why?

Because the people reading your Direct Mail pitch have 8,257 things to do and need your help to get this task completed correctly.

I have an even better way to improve your CTA. Every time I use it, response rates increase. You can get more orders, more leads, better average orders whenever you use it. Do you want to know what it is?

Then just do this:

a) Write an email to me: PTB@novuscom.net

b) In the Subject Line write: CTA Secret

c) Hit the send button. You don’t even need to put anything in the email message body – I’ll know what you want.

2. Don’t forget your suppliers.

When you test, make sure you include your printer and your list broker in your discussions.

First, they can be a font of helpful information. They can offer suggestions on production shortcuts, new list enhancements, even new lists to include in your test matrix.

Too often I see Direct Mailers who hand their printer a Fait Accompli. Then it’s up to the printer to figure out how to get this new piece or element created. They might run into postal regulations that hinder (or prohibit) your test from getting into the postal system.

Same thing with List Brokers.

When you want to do an acquisition A/B test, talk with your List Broker early in the process. They are a wealth of knowledge in the list business.

Here’s a quick example: I wanted to test different price points for a product. My broker suggested I test three lists built on direct mail responders with different higher average orders. We learned that a list with people with higher average orders worked best with a higher price point test, and those with a lower original dollar value preferred the lower ask. For the roll-out we offered higher products to some lists and lower price points to others.

3. Don’t forget to test your test.

Yup. You read that right.

Test your test.

Create a few mail-ready mock-ups of your test, especially if you are testing a new format (postcard vs. closed envelope) or radical size change (5” x 7” postcard vs. 8.5” x 5.5” postcard.)

Drop your new test into the postal system to your seed addresses.

Make sure they are deliverable. Check the timing of the delivery (in case your planned test is time-dependent.)

And meet with your internal order-processing team so that they understand the new test.

Here’s another war story: we did a test of 20,000 pieces. But the reports from mail opening and order entry departments showed no response. Zero. Zip.

I went down to the mail opened room to see what happened.

The supervisor said they didn’t know about the test, but that she had five trays of mail she had never seen before, and she thought they were misdirected mail from the competition!

The test was a great success, but we only knew about it weeks – not the usual days – after the test was mailed!

4. Don’t forget to validate your test.

You regularly mail 200,000 pieces per month.

You decide you want to do an A/B test of the envelope copy.

You create an A segment of 5,000 pieces with the new copy.

Then, you measure that response against the rest of the mailing.

WRONG!

Create your A segment of 5,000 pieces with the test envelope copy.

Create a B segment of 5,000 pieces with your control copy. (This is called the Test Control.)

Mail the rest of the 190,000 pieces of your control.

Make sure each segment has its own tracking code.

Measure A versus B… not A versus the full mailing.

This is critical, as it will validate the test – response from 5,000 against 5,000, not 5,000 against 195,000.

The test control is especially required when doing multiple testing.

Your matrix should have several Test Panels (A, B, V, etc.), one Test Control Panel (TC) the same size as the Test Panels, and the Control Panel (CP) with the remainder of your roll-out.

Measure the Test Panels against the Test Control for 100% accuracy and validation of your test.

4.(b) Don’t forget to retest.

Great! Test B is the winner! It beat all the other Test Panels and the Control Panel.

So you are going to roll out with that winner, right?

WRONG!

Re-test.

Re-test a larger segment.

If the test was 5,000 pieces, re-test your winner with 25,000 pieces.

If your test was 1,000 pieces, re-test with 5,000 pieces.

It’s just as you would do with a new list test. You test the minimum, then, if the list works, retest 5x the minimum. If that re-test works, you mail as much as you can afford!

5. Don’t forget all the order options.

Yes, you are a Direct Mailer. Yes, you have a great website or retail location you want to drive responders to.

But, don’t forget that there are many ways to respond.

Give options: a web address. An email address. A phone number. A physical location. A PO Box for mail in response.

The rule of thumb is that every direct mail element (except the outgoing envelope) should list 3 ways to respond: a phone number, a URL, and an address (email or physical.)

Oh, and if you use a QR code, congratulations and welcome to the 21st century.

But if your QR code does not include an actual URL that your reader can type in themselves, then go back to the last century and try to catch up later.

To Sum Up…

Direct Mail testing must be part of your plans.

But plan out your tests before you even start.

Every test should answer one simple question.

All you have to do is determine that question.

Is it more orders, more leads, higher value orders?

Does the boost come from a new headline, a new price point, a new order form layout?

Every element of your direct mail promotion is testable.

For one client, we did so many tests, we were down to testing whether blue-buff tinted paper or grey-buff tinted paper did better. (Blue won.)

So… test. Test smart. Test correctly. And don’t F-Up your test and waste all that time, money, and energy on a test that you did wrong!

Feel free to comment below.

Peter T. Britton

Idea Generator. Wordsmith. Resultant

PTB@novuscom.net