Why You Can’t Split Test Email Deliverability

After a conversation on Facebook, I am going to explain WHY Email Deliverability can't be measured with typical split testing. That is not to say elements that affect delivery cannot be tested just like any other email variable. But split testing email platforms is just not possible. Here's Why-

Image of what split testing email delivery is like

"Because at the end the day, you will end up with a basket of apples, oranges and cucumbers, with some rotten tomatoes mixed in."

So what is delivery and deliverability?

Email delivery means your Autoresponder or SMTP provider sent the email and it was accepted.

Email deliverability hopefully means you did not go to the spam folder, but Inboxed instead.

So let's explore how a deliverability split test is flawed at it's core.

Defining our fantasy split test

That would mean taking your list, splitting it into say two segments and then mailing each half, through two different email providers.

Half the list goes to Autoresponder Y or SMTP provider X.

For now we will call them X-mailer and Y-mailer.

So let's dig deep on factors that would skew an email deliverability split test, shall we?

All things must be equal:

Obviously you need the same from address, subject line and body copy and links.

But here are a few things you can't control:

Simple things like the HTML template you use at a particular email provider.

Or what IP address your email goes out on that day, when using typical mailing pools at most Autoresponders.

Could be good, could be bad, but you have no control here.

The reputation of your current list host. Because if you are mailing every week on X-mailer?

Then sign up to magic bullet of the week Y-mailer?

I can tell you now, Y-mailer is either going to greatly out-perform X-mailer, or massively under-perform.

And your first mailing through your new magic bullet, Y-mailer is going to throttle you on your first mailing.

So before you can test two, you would have to be mailing from both for a time.

So how are you going to measure these apples, oranges and cucumbers?

First problem is you are judging a split test by opens and clicks

So you are going to be judging success or failure of a both halves of your list by open and click rates.

This will be dependent on:

  • Engagers (opened or clicked in 60 days)
  • Non-engagers (no opens or clicks for 4 to 6 months)
  • Super-engagers (multiple opens and clicks, last 30 days)

But how can you evenly distribute Engagers, Non-engagers and Super-engagers evenly across both list segments?

Now let's say you take the time to segment your list into 3 sets of engagement, that's not that hard these days.

Here's your next problem...

Let's start with the big dawg, Gmail

So let's say you have engagement segmented?

Now you need to split that list into equal segments of Gmail.

You have to send equal parts Gmail to both X-mailer and Y-mailer platforms.

50% to X-mailer, 50% to Y-mailer.

Then go on to do the same with Yahoo, Microsoft, AOL, Cable ISPs and Domains.

Unless you can control all of that data accurately, then you can't split test deliverability.

Oh wait, did I tell you to scrub the list for spam traps first?


"Ah yeah... I forgot to tell you that." -So now back to point A- "Yeah, sorry about that..."

If you have spam traps on your list, and Segment A sends even one important trap to email platform Y-mailer?

Well then you have just destroyed the split test and you never even knew this happened.

"Because removing Non-engagers is not removing spam traps"

Bottom line on Email deliverability split testing

Jumping from service to service hoping for a magic bullet instead of putting in the proper time and work? Its a get rich quick mentality.

You would be much better off building a great reputation on your platform of choice.

Instead of expecting to load up your leads, then blast away on a cheap plan at just any email platform?

Why not take the two weeks to build a great reputation on a Dedicated IP address?

Or invest in your business and your delivery by paying just a little more for a great Pool IP?

And at Send13 we can help with that.

Unfortunately other Y-mailer platforms are not quite as helpful.

Because at the end of the day- Marketers are welcome here! At SEND13...

Comments are closed.