A/B testing in direct mail: how to find what actually works.

Most direct mail campaigns get sent once and never tested. A version goes out, results come in, and the conclusion is either "direct mail works" or "direct mail does not work for us." Neither is particularly useful.

What tends to get missed is the question underneath: which version of the campaign worked? Was it the headline? The offer? The format? The image on the front? Without testing, there is no way to know. This means the next campaign is built on the same guesswork as the last one.

A/B testing fixes this. It is not complicated, and you do not need a big budget or a data team to run one.

What is A/B testing in direct mail?

An A/B test splits your recipient list into two groups. Group A gets one version of your mailer. Group B gets a different version, with one thing changed. You measure the results, see which performed better, and use that to inform the next send.

The critical word is one. Change one variable at a time. Swap the headline, the image, and the offer all at once and you will not know which change moved the needle. Clean tests need clean variables.

Common things worth testing in direct mail:

• Headline or opening line: different angles, different lengths, direct versus curiosity-led
• The offer: percentage discount versus fixed amount, free gift versus free delivery, urgency versus no urgency
• Format: postcard versus letter versus greetings card, size, landscape versus portrait
• Call to action: QR code versus URL versus phone number, "shop now" versus "claim your discount"
• Image: product shot versus lifestyle, people in the image versus no people

Pick one. Run the test. Then pick the next one.

What is the goal of your test?

Before asking how to run an A/B test, it is worth asking why you are running one. The answer changes everything.

If the goal is to find out whether direct mail works for your business, a small test is unlikely to answer that question, and may actually give you a misleading result. If the goal is to find out which message, design, or call to action works best within a campaign you are already committing to, that is a different question entirely. It has a method, and it has a right answer.

Be honest about which question you are asking before you design the test. If budget pressure is influencing how you approach this, it's worth reading: This is the wrong time to cut direct mail.

How large should a direct mail A/B test be?

This is where most tests go wrong. A test group that is too small does not just produce uncertain results; it can actively mislead you.

Here is why. Based on Stannp.com's analysis of five million mail items, a good-to-great QR engagement rate sits between 3% and 5%. Take the top end of average - 3.55% - and a campaign of 10,000 recipients might return around 350 engaged responses. Those 350 people are distributed across your entire list.

Now send a test to 500 recipients: 5% of your total data. The statistical likelihood of that small group containing enough of those 350 engaged recipients to produce a meaningful signal is low. The result you get may say more about which 500 people you happened to pick than about which version of your creative actually works.

Our recommendation: if you are testing before sending to your full list, commit to at least 50% of your total data. Split it randomly. Anything less and the numbers are too thin to trust.

How do you set up a direct mail A/B test?

Here's how to structure a test that will actually tell you something:

1. Decide what you are measuring.

Response rate? QR code scans? Coupon redemptions? Decide before you send. A clear metric keeps you honest when results come in.

2. Split your list randomly. Unless you know your segments.

Random splitting works well when you do not have detailed customer profile data. If you do know your audience - different segments respond to different formats or messaging - use variable content to align recipients before upload, connecting each group with specific CTAs, images, QR codes, and dynamic landing destinations. An intentional split based on known customer behaviour is more useful than a random one.

3. Create your two versions.

Change one thing only. Both versions should go out at the same time to the same types of recipient.

4. Track responses separately.

Use unique QR codes, promo codes, or landing page URLs for each version. If both groups use the same code, you cannot tell the results apart.

5. Give it enough time.

Direct mail response windows are longer than email. Wait at least two to three weeks before drawing conclusions; longer for higher-value purchases.

6. Apply the winner.

Use the better-performing version as your new baseline. Then test the next variable against it.

Over time, this compounds. Each test improves the baseline. Campaigns get measurably better rather than staying flat send after send.

Which industries use A/B testing in direct mail?

A/B testing works across almost every sector. It tends to be most common in retail, eCommerce, financial services, real estate, automotive, hospitality, healthcare, home improvement, and SaaS, but any business trying to improve customer response rates can benefit from it. The mechanics are the same regardless of industry: one variable, two groups, honest measurement.

How do you interpret direct mail A/B test results?

Be honest about them. Sometimes the gap is clear. One version outperforms the other by a meaningful margin and you have a confident answer. Other times the difference is small enough that it could just be noise rather than a real insight.

If results are close, that is useful too. It tells you this particular variable does not matter much to your audience. Stop spending energy on it and test something else.

The goal is not to run tests for their own sake. It is to build up a picture of what your specific audience responds to, with evidence you can actually trust. Businesses that do this consistently end up with direct mail that performs better than those that do not. Not because of bigger budgets or more creative ideas, but because they know what works for their customers specifically.

If you are not testing yet: pick one campaign, split it in two, change one thing, and see what happens. That is the whole starting point.

Already a Stannp.com customer? Log in and get your first test running through automations. Not signed up yet? It only takes a few minutes to get started. 

Frequently asked questions.

What is A/B testing in direct mail?

A/B testing in direct mail means sending two versions of a mailer to separate groups of recipients, with one element changed between them. By tracking which version generates more responses, you can identify what your audience responds to and apply that to future campaigns.

What should the goal of a direct mail A/B test be?

That depends on what question you are trying to answer. Testing to find out whether direct mail works at all requires a large enough audience to produce statistically meaningful results. A small test group is unlikely to give you a reliable answer. Testing to find out which message, design, or CTA works best within a campaign you are already committing to is a more precise question with a clearer method. Be clear on which you are asking before you design the test. 

How many people do I need to run a direct mail A/B test?

More than most people assume. Based on Stannp.com's analysis of five million mail items, a good engagement rate sits around 3-5%. On a 10,000-item campaign that is roughly 350 engaged recipients spread across your entire list. A test to 500 people - 5% of the total - is statistically unlikely to capture enough of those engaged recipients to produce a reliable result. As a rule of thumb, commit to at least 50% of your total data before splitting into test groups. Anything less and the numbers are too thin to draw confident conclusions from. 

What should I test first in a direct mail campaign?

Start with the element most likely to affect whether someone acts on your mail. The offer or call to action usually has the biggest impact on response rates. Once you have a winning offer, test the headline. Then the format. Work through variables one at a time.

Can I test more than two versions at once?

Yes, this is sometimes called multivariate testing. However, for most businesses running direct mail, starting with straightforward A/B tests (two versions, one variable) produces cleaner results and simpler decisions. Add complexity once you have a process in place.

How long should a direct mail A/B test run?

At minimum two to three weeks. Direct mail works on a longer response cycle than email, and some recipients act on mail weeks after receiving it. Cutting the test window short risks drawing conclusions before the full picture is in.

What is the biggest mistake businesses make with direct mail A/B testing?

Testing too many variables at once. If you change the headline, the design, the offer, and the audience in the same test, you cannot know which change drove the result. Every additional variable you introduce makes the data harder to read. Keep it to one change per test.