How to Run a Holdout Test You Can Actually Read

A follow-up email either brings people back or it does not, and you only know which by leaving a random group untouched. Here is how to size it, run it, and read it without fooling yourself.

Read

9 min read

Published On

Sep 22, 2026

Read: 9 minutes

Last updated: September 22, 2026

In this guide

Why a holdout beats before-and-after, and what it needs to be fair
How many people each group needs, with a table you can reuse
How long to run it and what to measure
A worked example, and why 12 against 26 tells you nothing

You send follow-up emails to applicants who stalled. Some of them come back and finish. The question that matters is how many would have come back anyway. Many people finish an application a day or two late without any reminder. If you count every completion after an email as the email's work, the email will always look good.

A holdout test answers the question. You pick a group at random, send them nothing, and compare. The difference between the groups is the email's effect, give or take chance. It only works if you do four things properly: randomise, size it, wait long enough, and count the right outcome.

Why Before-and-After Fails

The tempting shortcut is to switch on follow-ups and compare this month's completion rate to last month's. It fails because everything else moved too. A promotion ran, a form changed, the month had a payday in it. You cannot separate the email from all that.

A holdout runs both versions at the same time, on the same kind of people. Whatever moves the whole funnel moves both groups. Only the email differs.

Randomise, Do Not Choose

Assign each stalled applicant to follow-up or holdout by chance. Do not assign by alphabet, by who signed up first, or by who a sales rep thinks is promising. Any rule a person can choose will leak a difference between the groups, and that difference gets counted as the email's effect.

List every applicant who stalls in a given week, with an ID for each.
Give each a random number. A spreadsheet's random function is fine.
Sort by that number. Put the first 10% to 20% in the holdout and the rest in the follow-up group, or split them evenly if your volume is small.
Freeze the assignment. Paste the result as values so it cannot reshuffle, and never move anyone between groups.
Keep the holdout group on whatever they get today, which may be nothing. Do not add a different nudge for them.

With volume, a 10% to 20% holdout is usual, so few people go without help. With low volume, an even split needs the fewest people. For the sheet itself, see Follow Up Stalled Applicants With a Google Sheet.

How Many People Each Group Needs

The number you need depends on three things: your baseline completion rate, the smallest improvement worth detecting, and how sure you want to be. The usual choices are a 5% false-alarm rate (a 95% test) and an 80% chance of catching a real effect.

For two equal groups comparing completion rates p1 and p2, the people needed in each group is about:

n per group = 2 x (1.96 + 0.84)^2 x p x (1 - p) / (p2 - p1)^2
(p = the average of p1 and p2)

The squared difference at the bottom is why small effects are expensive. Halve the lift you want to detect and you need about four times the people.

Baseline completion	Completion you hope for	People needed in each group
20%	30%	about 295
20%	25%	about 1,095
20%	22%	about 6,500
10%	15%	about 690
18%	27%	about 340

Read the second row. If 20% of stalled applicants usually finish and you want to catch a lift to 25%, you need roughly 1,100 people in the follow-up group and 1,100 in the holdout. Many platforms do not have that in a month, and that is not a reason to skip the test. It is a reason to pick the lift worth caring about or wait for more volume.

Why 12 Against 26 Tells You Nothing

Small tests feel reassuring and are not reliable. Suppose 26 applicants get a follow-up and 12 are held out. Nine of the 26 finish (35%). Three of the 12 finish (25%). The email looks ten points better.

It is not a result. With groups this size, the 95% margin of error on each rate is huge: the 35% could be anything from about 19% to 54%, and the 25% from about 9% to 53%. The ranges overlap almost completely. A standard exact test (Fisher's) on these counts gives a p-value of about 0.7, meaning a gap at least this large would appear by chance roughly seven times in ten even if the email did nothing.

The trap. If you run a tiny test and the email wins, you will believe it. If it loses, you will blame the sample. Either way you learned nothing. Decide the sample size before you start and do not stop early.

How Long to Run It

Run until you reach the sample size from the table, not until it looks significant.
Cover at least two full weeks, so every weekday appears twice. Behaviour on a Monday and a Friday differs.
Then wait out the lag. If most people who finish do so within 14 days of stalling, give the last people in the test 14 days after their email before you count.
Look at the result once, at the end. Checking every day and stopping at the first good-looking number inflates false alarms.

What to Measure

Pick one main outcome before you start and write it down. For stalled applicants it is usually the share who submit the application within a fixed window, such as 14 days after stalling. Count both groups the same way, from the same start date, whether or not the follow-up group opened the email.

Two traps. First, do not compare only people who opened the email to the holdout. People who open are already the more motivated ones, so the comparison flatters the email. Compare everyone assigned to follow-up against everyone assigned to the holdout, a method called intention to treat. Second, track a guardrail such as unsubscribes and complaints so a winning email does not quietly annoy people.

A Worked Example With Rosa's Bakery

Reading a holdout on applicants like Rosa's Bakery (illustrative numbers)

Rosa's Bakery is a single-location bakery with about $38,000 a month in card deposits. It is the kind of applicant who stalls at the ID step. Imagine a platform where 2,400 applicants like Rosa stall in a quarter.

Baseline: about 18% of such applicants finish within 14 days with no follow-up. The team wants to catch a lift to 27%. The table says about 340 per group, so 680 in total, comfortably within 2,400.

Setup: 700 applicants are randomly assigned, 350 to a follow-up and 350 held out. The rest of the stalled applicants are handled as usual.

Result after 14 days: 126 of 350 finish in the follow-up group (36%). 63 of 350 finish in the holdout (18%).

Reading it: the holdout shows what happens with no email, 18%. The follow-up group is 18 points higher. With 350 people per group, the 95% margin of error on each rate is roughly 4 to 5 points, so the gap is clearly real. Whether it pays for the effort is a separate question about what each completed application is worth.

These numbers are invented to show the arithmetic. They are not a benchmark. Your baseline and lift will differ.

When You Cannot Reach the Sample Size

Small platforms cannot always reach a few hundred per group. Aim for a bigger effect, which is cheaper to detect, by testing a real change such as a new first sentence. Run for longer. Or skip the test and say that you did: send the follow-up, track completion, and treat the number as a hint. What you should not do is run 12 against 26 and quote the winner.

To find the step where applicants stall before you test what to say to them, start with Find the Step Where Applicants Stall. For what the message should say, see What a Stalled-Applicant Email Should Say.

Common Questions

What is a holdout group?

A random group that does not get the thing you are testing. Comparing them to the group that does is the cleanest way to measure what the thing actually changed.

Is it fair to leave applicants without a follow-up?

People can still finish on their own, and it is often what they get today. Keep the group small when you have the volume, and end the test once you have an answer.

What if the result is not significant?

It means the test could not tell the groups apart, not that the email does nothing. Check whether you reached the planned sample size. If you did, the effect is smaller than the lift you set out to detect.