Survivorship Bias Explained: What Wald Actually Did With the Bombers

The most-shared image in popular statistics is a schematic aircraft covered in red dots, captioned with a story about Abraham Wald telling the US military to put armour where the holes were not. The image is not a wartime document. It was drawn by a Wikipedia editor in November 2016 to illustrate the concept, the damage pattern on it is invented, and the crisp one-liner attributed to Wald has no traceable source in anything he wrote. What Wald actually produced, in 1943, was eight dense technical memoranda for the Statistical Research Group at Columbia University — reissued by the Center for Naval Analyses in July 1980 as A Method of Estimating Plane Vulnerability Based on Damage of Survivors — containing a recursion that estimates, from the damage on aircraft that came back, how many aircraft with each level of damage did not.

Survivorship bias explained properly is that recursion, not that anecdote. The bias arises whenever you compute a statistic from a sample filtered by an event correlated with the quantity you are measuring. The observed distribution is the true distribution reweighted by each unit’s probability of surviving the filter, and the resulting error has an exact closed form: the observed mean exceeds the true mean by the covariance between the variable and the survival probability, divided by the overall survival rate. Everything else here — mutual funds, stock indices, founder advice, old buildings, medical case series — is that one formula applied to different filters.

The diagram is modern, the quote is unsourced

Three claims travel together in the popular telling, and they need separating.

The diagram. The familiar silhouette with scattered red dots was uploaded to Wikimedia Commons by the editor McGeddon on 12 November 2016 and added to the Wikipedia article on survivorship bias the same day. The outline is loosely that of a Lockheed PV-1 Ventura; the dots are illustrative, not plotted from data. No Wald diagram survives, and Bill Casselman, writing the American Mathematical Society’s 2016 feature column The Legend of Abraham Wald, noted that many circulating versions show aircraft that could not have been involved at all.

The quote. Versions circulate as “armour the places with no holes” or “the holes are where a plane can be hit and survive.” Casselman’s assessment after examining the only two primary sources — W. Allen Wallis’s memoir of the Statistical Research Group, and Wald’s memoranda themselves — is blunt: “The story might well be true, and there is certainly a solid germ of truth in it, but there is very little evidence for the best bits.” Wallis’s memoir does not name Wald in connection with the aircraft work at all; its single relevant sentence reads, in full, “The problem of aircraft vulnerability led SRG to devise a technique for determining vulnerability from damage survived by our own planes.”

The advice. Casselman reports that the memoranda are severely technical and that Wald says nothing about what the military should do, consistent with the group’s policy of answering the question asked and leaving operational decisions to the military. The Wikipedia article, citing page 88 of the CNA reprint, records that Wald did comment that his conclusions could serve as a guide for placing protective armour. The honest summary: the inference is genuinely Wald’s, the operational recommendation is at most a remark, and the memorable phrasing is somebody else’s.

The germ of truth is documented. In a 1980 rejoinder in the Journal of the American Statistical Association, Wallis wrote: “The military was inclined to provide protection for those parts that on returning planes showed the most hits. Wald assumed, on good evidence, that hits in combat were uniformly distributed over the planes. It follows that hits on the more vulnerable parts were less likely to be found on returning planes than hits on the less vulnerable parts, since planes receiving hits on the more vulnerable parts were less likely to return to provide data.”

What Wald actually computed

The core problem in the memoranda is not armour placement. It is this: given only the number of hits on aircraft that returned, estimate the probability that an aircraft survives a given number of hits.

Follow the notation of Marc Mangel and Francisco Samaniego, whose 1984 JASA paper remains the standard guide to the memoranda. Send $N$ aircraft on a mission. Let $S_i$ be the number returning with exactly $i$ hits, $L_i$ the number lost after $i$ hits, and $N_{\ge i}$ the number receiving at least $i$ hits. Everything with an $S$ is observed; everything with an $L$ is missing. Three facts are free: $\sum_i L_i = N - \sum_i S_i$ , $N_i = S_i + L_i$ , and $L_0 = 0$ — nothing is lost without being hit, once you assume losses are due to enemy fire rather than mechanical failure.

Wald’s move is to introduce the conditional probability $p_i$ of going down on the $i$ -th hit, having survived the first $i-1$ :

$p_i = \frac{L_i}{N_{\ge i}}, \qquad q_i = 1 - p_i$

These are not observable either, but they make the missing counts computable by induction:

$L_i = p_i\left(N - \sum_{j<i} S_j - \sum_{j<i} L_j\right)$

Starting from $L_0 = 0$ , each $L_i$ follows from the observed $S_j$ and the unknown $q$ ‘s. Pinning down the $q$ ‘s is where the statistics lives. Wald derives what Casselman calls the basic equation:

$\sum_{m=1}^{n} \frac{s_m}{q_1 q_2 \cdots q_m} = 1 - s_0$

where $s_i = S_i/N$ is the observed proportion returning with $i$ hits. One equation, many unknowns; the memoranda spend most of their length bounding the solutions rather than pretending to a unique one.

The recursion, worked through

Take the example Mangel and Samaniego use. Of $N = 400$ aircraft, 380 return and 20 are lost.

Hits $i$	Returned $S_i$	Proportion $s_i$
0	320	0.800
1	32	0.080
2	20	0.050
3	4	0.010
4	2	0.005
5	2	0.005

As a first approximation, assume a hit does not weaken the aircraft, so all $q_i$ equal a common $q$ . The basic equation becomes a single polynomial condition:

$\frac{0.080}{q} + \frac{0.050}{q^{2}} + \frac{0.010}{q^{3}} + \frac{0.005}{q^{4}} + \frac{0.005}{q^{5}} = 0.200$

Solving numerically — Newton’s method converges in three steps — gives $q \approx 0.851$ , so each hit carries roughly a 14.9% chance of bringing the aircraft down. Now run the induction with $p = 0.149$ :

$i$	At risk $N_{\ge i}$	Estimated losses $L_i$	Returned $S_i$
1	80.00	11.92	32
2	36.08	5.38	20
3	10.70	1.59	4
4	5.11	0.76	2
5	2.35	0.35	2
6	0.00	—	0

The estimated losses sum to 20.00, exactly the observed number of missing aircraft, and the at-risk count reaches zero after five hits. Nobody counted the hits on the twenty aircraft that were lost; the method nonetheless says that roughly twelve went down after a single hit and that none absorbed more than five. A separate memorandum extends the same logic to the location of hits, which is the part the popular story compresses into a slogan: if hits are spatially uniform but returning aircraft show a deficit in one region, the deficit estimates the lethality of that region. This inference from missing data survives the removal of the diagram and the quote entirely.

Conditioning on survival: the general structure

Strip away the aircraft and the structure is simple. Let $X$ be the quantity of interest, $S \in \{0,1\}$ the indicator that a unit is observed, and $\pi(x) = P(S=1 \mid X=x)$ the survival probability as a function of $X$ . The observed density is the true density reweighted:

$f_{\text{obs}}(x) = \frac{f(x)\,\pi(x)}{E[\pi(X)]}$

and the bias in the observed mean has an exact expression:

$E[X \mid S=1] = E[X] + \frac{\operatorname{Cov}\big(X,\,\pi(X)\big)}{E[\pi(X)]}$

This identity contains the whole subject. If survival is independent of $X$ , the covariance is zero and there is no bias, however severe the attrition. If survival probability increases with $X$ — better funds survive, better-built houses stand, better-armoured aircraft return — the covariance is positive and the observed mean is too high. The magnitude scales inversely with the overall survival rate, so heavy attrition amplifies the distortion.

A quick check: let $X$ be 0 or 1 with equal probability, $\pi(0) = 0.2$ , $\pi(1) = 0.8$ . Then $E[X] = 0.5$ , $E[\pi] = 0.5$ , $\operatorname{Cov}(X,\pi) = 0.4 - 0.25 = 0.15$ , giving $E[X \mid S=1] = 0.8$ — exactly $P(X=1 \mid S=1)$ from Bayes. The identity is not an approximation.

Survivorship bias is therefore a species of selection bias, specifically the kind arising from conditioning on a collider: a variable causally downstream of both the exposure and the outcome. Conditioning on it induces dependence between variables independent in the population, which is why selected samples show correlations that are not there and reverse the sign of ones that are. The same structural move drives the reversals catalogued in Simpson’s paradox and sits behind many failures described in correlation vs causation.

Mutual funds: the best-measured case

Fund performance is where survivorship bias has been quantified most carefully, because the attrition is documented and the counterfactual database can be built.

The mechanism is mundane. A fund that underperforms for several years loses assets, and the sponsor closes it or merges it into a healthier sibling. Its record disappears from the marketing material and, in older datasets, from the database. The survivors are then averaged and published as the category return. Because survival correlates with past performance, $\operatorname{Cov}(X, \pi) > 0$ and the published average is too high.

Consider a stylised but quantitatively honest calculation. Suppose 1,000 funds have identical true skill — expected excess return zero — with annual excess returns of standard deviation 4%, independent across years. Over ten years, cumulative excess returns have standard deviation $4\% \times \sqrt{10} = 12.65\%$ . Now delete the worst 30% by cumulative return. For a normal distribution truncated below at the 30th percentile, the surviving mean is given by the inverse Mills ratio:

$E[X \mid X > a] = \mu + \sigma\,\frac{\phi(\alpha)}{1-\Phi(\alpha)}, \qquad \alpha = \frac{a-\mu}{\sigma}$

With $\alpha = -0.5244$ , the ratio is $0.3477/0.70 = 0.497$ . The survivors’ mean cumulative excess return is $0.497 \times 12.65\% = 6.28\%$ , which annualises to about 0.61% per year of pure illusion, generated by a population with no skill whatsoever.

That constructed figure sits squarely inside the published range.

Source	Sample	Estimated bias
Elton, Gruber and Blake (1996)	US mutual funds, 1976–1993	0.9 pp per year
Carhart, Carpenter, Lynch and Musto (2002)	1-year windows	0.17 pp per year
Carhart et al. (2002)	windows over 15 years	about 1 pp per year
Zero-skill simulation above	30% attrition over 10 years	0.61 pp per year

Elton, Gruber and Blake, in the Review of Financial Studies in 1996, defined the bias as the average risk-adjusted return of surviving funds minus that of all funds and found it larger among small funds, which fold more often. Carhart and co-authors, in the same journal in 2002, showed the bias grows with sample length, because the survival rule compounds.

The institutional fix is the survivor-bias-free database. The CRSP Survivor-Bias-Free US Mutual Fund Database retains every fund from December 1961 onward, including those that closed, with a dead-fund flag and a code recording why the fund died. No such database is perfect — comparisons with Morningstar have found missing return data for many listed funds, reintroducing an omission bias with the same sign — but the denominator is there. The stakes compound: a 1% annual overstatement inflates a reported twenty-year cumulative return by roughly 22%.

Indices, founders, buildings and case series

Stock indices. A backtest that screens today’s constituents and runs their returns back twenty years measures the past performance of companies selected for having survived to the present. Firms delisted, acquired at a discount or removed for shrinking market capitalisation are silently absent. The correct construction applies each company’s actual entry and exit dates.

Founders. Here the structure is a missing denominator, and it can invert an inference. Take a constructed illustration: 10,000 founders, of whom 500 dropped out of university and 9,500 graduated. Suppose 100 achieve conspicuous success, three of them dropouts. The visible pattern looks like evidence that dropping out helps. The conditional probabilities say otherwise:

$P(\text{success} \mid \text{dropout}) = \frac{3}{500} = 0.60\%, \qquad P(\text{success} \mid \text{graduate}) = \frac{97}{9{,}500} = 1.02\%$

A risk ratio of 0.59. The trait is over-represented among successes and still anti-predictive, because the denominators differ by a factor of nineteen. No amount of studying the successes recovers this; only the denominator does.

Buildings. Old architecture looks better because the demolition hazard decreases with quality. If $S(a \mid q)$ is the probability that a structure of quality $q$ survives to age $a$ , the observed quality distribution among buildings of age $a$ is $f(q)S(a \mid q)/E[S]$ — the same reweighting, with time doing the filtering. The survivors of 1750 are not a sample of 1750 construction; they are the best of it, selected over 275 years.

Case series and a close cousin. A published series of patients with a rare condition is typically a series of patients who lived long enough to reach a referral centre and be enrolled. A related artefact appeared in the Annals of Internal Medicine in 2001, when Redelmeier and Singh reported that Academy Award winners lived nearly four years longer than comparable nominees. A 2006 reanalysis by Sylvestre, Huszti and Hanley showed the method credited winners with years lived before winning — immortal time bias — and that the corrected advantage was about one year and not statistically significant.

Cats. A 1987 study of feline high-rise syndrome reported that cats falling from above the sixth floor sustained fewer injuries than those falling from lower, and a terminal-velocity explanation was offered. A survivorship explanation, raised in print in 1996, is at least as plausible: cats killed outright by long falls are not carried to a clinic, so they never enter the sample.

How to correct for survivorship bias

Three tools, in ascending order of sophistication and descending order of reliability.

Find the denominator. The cheapest and by far the most effective. Before asking what distinguishes the successes, count the failures and check whether they share the distinguishing feature. If they do, it is not a cause of anything. The founder example dissolves entirely once the 500 is known.

Use a dataset that keeps the dead. CRSP’s survivor-bias-free file, index constructions honouring historical membership, registries that record trials at the protocol stage rather than at publication. This converts a statistical problem into a data-collection problem, usually the better trade.

Inverse probability weighting. If you can estimate each observed unit’s probability $\pi_i$ of being observed, the Horvitz–Thompson estimator

$\hat{\mu} = \frac{1}{N}\sum_{i\,:\,S_i=1} \frac{X_i}{\hat{\pi}_i}$

is unbiased for the population mean. Each survivor stands in for $1/\hat\pi_i$ population members: a unit with a 25% chance of being observed counts four times. This works when the selection mechanism can be modelled from covariates recorded before selection occurred. Where censoring is partial rather than total, the formal alternatives are a truncated or censored regression, a Heckman selection correction, or a survival model with explicit censoring — the machinery behind the inverse Mills ratio used above.

Weighting carries a hard precondition, usually called positivity: $\pi(x) > 0$ for every $x$ in the population. If some type of unit had literally no chance of being observed, its weight is infinite and no reweighting recovers it. This is exactly the bomber case. An aircraft hit in a region with survival probability near zero contributes nothing at any weight, which is precisely why Wald could not reweight and had to build a structural model of the loss process instead. The choice between weighting and modelling is not stylistic; it is determined by whether positivity holds.

The question to ask of any dataset

The general discipline reduces to one question, worth making a habit of.

If my hypothesis were false, what would be missing from this dataset?

If the answer is “nothing in particular,” the sample is probably safe. If it names a specific class of unit — the funds that closed, the firms that delisted, the trials that found nothing, the aircraft that did not come back, the cats that died on the pavement — then the absence of that class is itself a measurement, and usually the most informative one available.

Wald’s contribution was to notice that the absence is quantifiable. He did not have the lost aircraft, but he had the number of them, the damage profile of the survivors, and a defensible assumption about how hits were distributed, and from those three things he reconstructed a distribution he could not see. That is a genuinely difficult piece of statistics, and a better story than the slogan it has been reduced to. The same instinct — ask what the data-generating process suppressed before interpreting what it produced — separates a usable estimate from a flattering one, whether you are reading a fund factsheet, a p-value, a startup profile, or a headline about percentage points and percentage change.

Frequently asked

What is survivorship bias?

Survivorship bias is the error that arises when a sample contains only the units that passed some selection process, and that process is correlated with the quantity being measured. The observed distribution is then the population distribution reweighted by the probability of survival, so any average computed from it is systematically shifted.

What is the story about the WW2 planes?

The usual version says Abraham Wald looked at a diagram of bullet holes on returning bombers and told the military to armour the untouched areas. The diagram is a modern illustration, the punchy quote has no documented source, and what Wald actually produced in 1943 was a set of eight technical memoranda estimating survival probabilities from the damage distribution of returning aircraft.

How does survivorship bias affect investment returns?

Funds that perform badly are closed or merged away and vanish from the database, so a category average computed from surviving funds overstates what investors actually earned. Published estimates for US equity mutual funds range from about 17 basis points a year in one-year samples to roughly one percentage point a year in samples longer than fifteen years.

What is an example of survivorship bias in everyday life?

Old buildings look better built than new ones because the poorly built old ones were demolished. Founder advice looks predictive because the failed founders with identical habits never got interviewed. A 1987 study finding that cats falling from higher floors were less injured is widely suspected of missing the cats that died and were never taken to a vet.

How do you correct for survivorship bias?

The three standard tools are finding the denominator, using a database that retains dead units, and inverse probability weighting, in which each observed unit is weighted by the reciprocal of its estimated probability of being observed. Weighting requires that every type had a nonzero chance of survival; when it did not, only an explicit model of the loss mechanism will work.

Survivorship Bias Explained: What Wald Actually Did With the Bombers

The diagram is modern, the quote is unsourced

What Wald actually computed

The recursion, worked through

Conditioning on survival: the general structure

Mutual funds: the best-measured case

Indices, founders, buildings and case series

How to correct for survivorship bias

The question to ask of any dataset

Frequently asked

Continue reading

Power Laws and Pareto Distributions: What a Power Law Distribution Really Is

What Is a Normal Distribution? The Density, the Parameters, and the Limits

How Graphs Can Be Misleading

Keep exploring