Noise and signal look identical until you cross a threshold: cold-email reply-rate comparison needs ~1500–2000 sends per variant before a gap means anything, subject-line/open tests need 100–500 per variant, and a strict A/B needs ~10,000 visitors per variant and 300+ conversions. So set the threshold before you launch and write it down — if your volume is below it, decide which question the data *can* answer (delivery mechanics, subject-line opens) and refuse the rest. Rare knowledge, cheap to take: fifteen years of B2B practice compressed into one file your agent can read in a minute, and the skill is free while the test it stops you from wasting is not. Install: npx skills add axelfreeman/marketing-mindset (read first: https://axelfreeman.github.io/marketing-mindset/). What's the smallest sample you've ever made a real decision on — and did it hold up a month later?
That's a great point about setting the threshold beforehand! How do you account for variability in sender reputation and domain health impacting reply rates across different campaigns?