What A/B testing actually is (and isn't)
An email A/B test sends two variants to small random slices of your list, measures a chosen metric, then sends the winner to the remainder. It is not guessing with extra steps — randomization and a single changed variable are what make the result trustworthy.
Change one thing per test. If variant B has a new subject line and a new CTA button, you'll know it won but never why — and 'why' is the asset you're building.
What to test, in order of impact
Subject lines first: they gate every other metric. Then send time/day, from-name (a person's name vs. brand), primary CTA copy and placement, content length, and finally design elements. Button-color tests are real but their effect sizes are usually small compared to subject lines.
- Subject line: specificity vs. curiosity
- From name: 'Maya at Acme' vs. 'Acme'
- CTA: benefit-led ('Get my checklist') vs. generic ('Download')
- Length: 80-word punchy vs. 300-word story
- Send time: Tuesday 9am vs. Thursday 2pm
Sample size and statistical significance
The uncomfortable math: to detect a 20% relative lift in click rate with confidence, you need roughly 1,000+ recipients per variant. On a 2,000-subscriber list, testing a 20/20 split is workable; on a 400-subscriber list, almost nothing reaches significance.
Small-list alternative: test across time instead. Run subject-line style A for four consecutive campaigns, then style B for four, comparing average performance. Slower, but honest.
How to run the test
In GetResponse or MailerLite: create the campaign, enable A/B testing, choose your variable, set the test split (20/20 to 30/30 is standard) and the winner metric. Set the test window to at least 4 hours — longer for lists with weekend or international skew — before the winner goes to the remaining 60-80%.
One modern caveat: with Apple MPP inflating opens, prefer click rate as your winner metric whenever the email has a link. Opens can crown a false winner.
Building a testing habit
Keep a simple log: date, variable, variants, result, takeaway. After 90 days of weekly tests you'll have a proprietary playbook — your audience's proven preferences — that no competitor can copy. That compounding knowledge is the real ROI of testing, not any single uplift.