A/B Testing

Test multiple subject lines per step to find what resonates best with your audience. Cleanmails automatically picks the winner after enough data.

How It Works

1. Add multiple subject lines to any campaign step (stored as a JSON array)
2. Cleanmails distributes sends evenly across variants (round-robin)
3. Once each variant reaches 100 sends, the system compares open rates
4. The variant with the highest open rate becomes the auto-winner
5. All subsequent sends use only the winning subject

Setting Up A/B Test

In the campaign step editor, add multiple subjects:

[
  "Hey {{FIRST_NAME}}, quick question about {{COMPANY}}",
  "{{FIRST_NAME}} — saw your work at {{COMPANY}}",
  "Idea for {{COMPANY}}'s outreach"
]

Viewing Results

Check per-variant stats via the API:

curl http://YOUR_SERVER/v1/outreach/campaigns/ab-stats/CAMPAIGN_ID

Returns:

[
  {"subject_variant": 0, "sends": 150, "opens": 45, "clicks": 12, "replies": 5, "open_rate": 30.0, "click_rate": 8.0, "reply_rate": 3.3},
  {"subject_variant": 1, "sends": 148, "opens": 52, "clicks": 15, "replies": 8, "open_rate": 35.1, "click_rate": 10.1, "reply_rate": 5.4},
  {"subject_variant": 2, "sends": 147, "opens": 38, "clicks": 9, "replies": 3, "open_rate": 25.9, "click_rate": 6.1, "reply_rate": 2.0}
]

Auto-Winner Threshold

The auto-winner is selected when all variants have at least 100 sends. The variant with the highest open rate (excluding bot opens) wins.