It is a Wild West out there when it comes to testing for attrition in field experiments. Researchers use different tests, describe them with different terms, and somehow always find a way to conclude that attrition is not a problem in their case. So I was happy to read Testing Attrition Bias in Field Experiments by Dalia Ghanem, Sarojini Hirshleifer and Karen Ortiz-Becerra (Journal of Human Resources, 2026; ungated version). It turns out I was doing it wrong. Chances are, so are you.
The authors review 96 published field experiments with baseline data. The most common test — reported in 79 percent of the experiments with attrition above 1 percent — checks whether attrition rates differ between treatment and control. But equal rates are neither necessary for an unbiased estimate nor, without extra assumptions, sufficient: rates can differ while your estimate is fine, and identical rates can hide serious bias. The second most common test, in 61 percent, checks whether treatment and control respondents are balanced at baseline. Closer, but only half of the correct test.
Test 1: Is the comparison of treatment and control respondents unbiased, for the respondents? The paper calls this internal validity for respondents (IVal-R). The identifying assumption is that treatment is as good as randomly assigned conditional on response status. Its testable implication is that both columns are balanced at baseline, jointly: TR = CR and TA = CA. I used to test each separately.
Test 2: Is that effect also the effect for your original sample? The paper calls this internal validity for the study population (IVal-P). The testable implication is that all four cells are equal. I will call them the respondent test and the representativeness test.
In regression form, using the baseline outcome and the full baseline sample:
Y0 = α + βT Treat + βA Attrite + βTA Treat × Attrite + γ + ε,
where γ are strata fixed effects (more on those below); for clustered designs, standard errors are clustered at the unit of randomization. The respondent test is the joint test βT = βTA = 0. The representativeness test adds one restriction: βA = βT = βTA = 0. Strictly, the paper’s tests compare entire baseline distributions; the regression tests the difference in means, which is what most of us will run, and the two coincide for binary variables.
The authors provide a Stata command that does all of this: `ssc install attregtest`.
Why also test balance among attritors?
My instinct was that we only care about balance in the sample of respondents, since that is our estimating sample. That is wrong for two reasons. First, the assumption you need, treatment as good as random given response status, has implications for attritors too. Testing respondents only uses half of the testable implications. Second, the respondent-only test is often badly underpowered. Suppose your program keeps some weak students enrolled who would otherwise have dropped out and vanished from the sample. With low attrition, these program-induced responders might be 2 percent of your treatment respondents, far too few to detect in a balance test. But their counterparts in the control group are attritors, where they can be a large proportion of a small group.
What about stratified randomization?
If you randomized within strata, the authors discuss three ways to run the test. The first two work within each stratum, interacting everything with strata dummies: either a single test of all the within-stratum restrictions at once, or a separate test per stratum followed by a multiple-testing correction that controls the family-wise error rate (FWER) or the false discovery rate. The authors motivate the per-stratum version for the case where response problems hit some strata but not others: you stratified by region, say, and one region is remote or flood-prone, so your sample there is hard to reach. The third way is simpler: the regression above, with strata fixed effects and no interactions.
I have two concerns with the within-strata versions.
1. The fully interacted version over-rejects the null, badly so in clustered trials. Interacting the test with strata multiplies the number of restrictions you test at once: with ten strata, the representativeness test already stacks thirty. Tests with many restrictions are known to over-reject when standard errors are clustered (see MacKinnon, Nielsen and Webb 2023 and Kerwin, Rostom and Sterck). I asked Claude to run some simulations (posted on my website; I did not verify the code, but it is consistent with the theory). With 100 clusters, even four strata is too many for the test to hold its size, and with ten strata the within-strata representativeness test rejects 86 percent of the time when nothing is wrong. The same problem arises with heteroskedasticity-robust standard errors under individual-level randomization (Anatolyev and Sølvsten 2023), but there it is far less severe: in the simulations, it only starts to bite once strata fall below roughly 200 observations each.
2. Lower statistical power. Conversely, the fully interacted version has lower power because of its many degrees of freedom, and the FWER correction is conservative. That hands researchers a way to falsely conclude that attrition is not a problem. A second set of simulations, with individual-level randomization so that over-rejection is not the issue, shows that the ranking depends on the nature of attrition. If the attrition problem is diffuse — slight bias in every stratum — the fixed-effects test has the most power and the per-stratum approach with an FWER correction the least. The ranking flips when the violation is concentrated in one stratum, the case the per-stratum approach was designed for.
So I believe it is more sensible to just use the fixed-effects test. The authors agree: they propose it for exactly this situation (Appendix A.2 of the paper, equation 19), use it in their own empirical exercise whenever there are more than ten strata, and attregtest reports both versions. The trade-off is that the fixed-effects version tests an implication of the sharp restriction rather than the restriction itself — a price worth paying once the restrictions pile up.
