What We Miss About Missing Values

As the old adage goes, a wise man once said nothing at all. Unfortunately, the same reverence is rarely extended to missing data. A row with a blank cell is often treated as a problem to be solved before analysis can begin: drop it, fill it with an average, do whatever is easiest and move on. The blank cell is, by this logic, a defect in the record rather than a fact about the world. But the absence of a measurement can provide real insight, and how we handle it may dramatically alter the conclusions we draw. Missingness is better understood not as an unfortunate nuisance, but as a byproduct of the process that also generates the data we observe.In a clinical trial, the patients who drop out before the end-of-study assessment are not a random sample of those who enrolled. They are disproportionately the ones whose treatment failed, whose adverse events were severe, or whose underlying condition deteriorated. On a signup form, an optional income field is more likely to be left blank when the true figure is one the user would rather not disclose. In a product experiment, users who churn before the measurement window closes never produce the outcome that was meant to be recorded, and may be disproportionately likely to do so because of the treatment they received. In each case, the process determining whether something is observed may itself be related to the phenomenon we are trying to understand.The same concern applies to sensors. Their impartiality over human self-report is real in that they cannot refuse to answer an awkward question or forget to make a recording, but this is often overstated as a guarantee of unbiasedness. Whether a value is recorded at all depends on the operating conditions of the instrument, which can itself be shaped by the phenomenon it was built to measure. A PM2.5 sensor sends a laser beam through a sample of air, using measurements of the light scattered by particulates to estimate pollutant concentration. But the particles being measured can also affect the instrument itself: high particulate loading can degrade sensor performance and lead to outages (Safarov et al., 2025). In this case, gaps may be more likely during the high-pollution events the network was built to capture. This is an increasingly common pattern in our brave new "Internet of Things", with seismometers that clip during earthquakes, strain gauges that fail under heavy loads, and smart meters which drop outage reports precisely when a blackout is at its peak.Rubin (1976) formalises three mechanisms by which data go missing, expressed as constraints on the missingness probability P(R∣Yobs,Ymis)P(R \mid Y_{obs}, Y_{mis}), where Y is the variable of interest partitioned into observed values YobsY_{obs} and unobserved values YmisY_{mis}, and R is a binary indicator of whether a given value of Y is observed. Data are missing completely at random (MCAR) when P(R∣Yobs,Ymis)=P(R)P(R \mid Y_{obs}, Y_{mis}) = P(R): the probability of a gap is independent of everything else in the dataset. Under MCAR, dropping the incomplete rows does not bias an estimator, though it reduces the sample size and inflates the standard errors, so the loss is one of precision rather than of validity. Data are missing at random (MAR) when P(R∣Yobs,Ymis)=P(R∣Yobs)P(R \mid Y_{obs}, Y_{mis}) = P(R \mid Y_{obs}): the probability of a gap depends only on the values one does observe. Under MAR, unbiased estimation is possible in principle using methods that correctly condition on the observed variables driving missingness. Data are missing not at random (MNAR) when P(R∣Yobs,Ymis)P(R \mid Y_{obs}, Y_{mis}) depends on YmisY_{mis}; the gap encodes information about the very quantity one cannot see, and no method conditioning only on observed values can identify the unobserved distribution without further assumptions. The distinction between MAR and MNAR is fundamentally untestable from observed data alone: the definition of MAR depends on values we cannot see. Which regime applies must be argued for on grounds outside the data itself, from expert knowledge of the mechanism, features of the study design, or structural understanding of what makes the record incomplete. In the PM2.5 case, it is the physical understanding of how pollution conditions affect the monitoring process that can make MNAR a plausible mechanism, not any statistical test. Every method for handling missingness therefore begins from an assumption about the process that produced the gaps, an assumption the incomplete dataset cannot supply on its own.One response is to model the selection process directly. Suppose we want to estimate the value of a marketing offer using subsequent customer spending. Spending is only observed for customers who remain active long enough to make a purchase, and the factors that determine whether a customer remains active may also affect how much they would have spent. Simply analysing the customers who generated a transaction therefore selects on a process related to the outcome itself. Heckman (1979) formalised this class of problem by modelling the process that determines whether an outcome is observed. His canonical example was wages, which are only recorded for individuals who choose to work: if the same unobserved characteristics that influence the decision to work also influence wages, the observed workers are not a random sample of the population.Heckman's approach was to treat this selection process as part of the model. An outcome equation for Y is paired with a selection equation describing whether Y is observed, allowing the unobserved factors affecting selection and the outcome to be correlated. The original two-step estimator first models the probability that an observation appears in the dataset, then uses this information to correct the outcome equation for the fact that the observed sample is not random. This correction is captured by the inverse Mills ratio, λ(zi)=ϕ(zi)/Φ(zi)\lambda(z_i)=\phi(z_i)/\Phi(z_i), which comes from the expected selection bias induced by observing only outcomes that pass the selection threshold.The strength of Heckman's approach is also its limitation. The relationship between the unobserved factors driving selection and the outcome cannot be tested from the observed data alone. The correction works only because we have imposed a strong assumption about the hidden process generating missingness, namely that the unobserved components follow a joint normal distribution. Under the weaker MAR assumption, Robins, Rotnitzky and Zhao (1994) take a different approach. Rather than modelling selection on unobservables, they estimate the target parameter by reweighting the observed records according to their probability of being observed. Writing πi\pi_i for the probability that unit i is observed, the inverse probability weighting (IPW) estimator of the population mean is μ^IPW=n−1∑i(Ri/π^i)Yi\hat{\mu}_{\text{IPW}} = n^{-1}\sum_i(R_i/\hat{\pi}_i)Y_i. Each observed record is weighted by 1/π^i1/\hat{\pi}_i so observations that were unlikely to appear in the dataset contribute more heavily, representing similar observations that were not seen. The estimator is unbiased under MAR provided the observation probability model is correctly specified. Modern extensions, including augmented IPW (AIPW), combine this reweighting approach with an outcome model to obtain the doubly robust property: the estimator remains consistent if either the model for the outcome or the model for the observation probability is correctly specified.Rather than reweighting the observed records, another approach is to represent the uncertainty about the missing values directly. Multiple imputation replaces each missing value m times with draws from a model of plausible values, runs the analysis on each of the m completed datasets, and combines the results. The pooled point estimate is the average of the per-imputation estimates; the pooled variance is T=VˉW+(1+1m)VBT = \bar{V}_W + \left(1+\frac{1}{m}\right)V_B, where VˉW\bar{V}_Wis the average of the within-imputation variances and VBV_Bis the between-imputation variance of the point estimates across imputations. The (1 + 1/m) term accounts for the additional uncertainty introduced by estimating the missing values from only a finite number of imputed datasets. Because the true values are unknown, multiple imputation carries uncertainty about the missing values through to the final analysis rather than treating the imputed values as if they had been observed. Sterne et al. (2009) therefore emphasise that the imputation model must be specified carefully and that its validity depends on the assumptions about why the data are missing.Where the missingness mechanism cannot be identified from the observed data alone, sensitivity analysis asks how much the conclusion depends on assumptions about what was not seen. Tipping-point analysis, discussed by Yan, Lee and Li (2009), does this by perturbing the imputed values by a shift parameter δ\delta representing a departure from the MAR assumption, increasing δ\delta until the primary conclusion changes, and reporting the amount of departure required to overturn the result. Cro et al. (2020) caution that the range of plausible δ\delta values should be agreed before examining the result, because otherwise the analyst's judgement of what counts as plausible can unconsciously move towards whatever value reverses the conclusion. If the conclusion only changes under an implausibly large departure from MAR, the result is relatively robust; if a plausible departure is sufficient, it is fragile.The methods above also depend on what we are trying to learn from the data. Missingness is not an inherent property of a measurement; it depends on the question being asked. Consider an A/B test of a new checkout flow. If we want to know the effect of assigning users to the new version, then the outcome of every user assigned to it matters, including those who leave before reaching the checkout. If instead we want to know the effect of the new flow among users who actually use it, those who never reach it fall outside the population we are studying. These are different questions about the same experiment, and they can have different answers. The 2019 ICH E9(R1) addendum formalises this idea through the concept of an estimand: a precise specification of the quantity a study aims to estimate. Deciding how to handle missing data therefore starts with deciding what we want the analysis to tell us.The right approach also depends on the purpose of an analysis. Sperrin, Martin, Sisk and Peek (2020) argue that missing data should be handled differently depending on whether the goal is inference or prediction. In inference, we want to estimate an underlying relationship (such as a treatment effect or regression coefficient), and the challenge is avoiding bias caused by the missingness process. In prediction, the goal is different: the question is whether the model performs well on future observations. If missingness itself contains information about the process generating the data, then discarding that information can make predictions worse. Consider a model that scores sales leads by how likely they are to convert. Whether a lead has filled in optional fields, such as company size or budget, may itself be predictive, because leads who complete more of the form are often more serious. The pattern of which fields are blank could therefore carry information about conversion that a model imputing over the gaps would discard. However, this only works when the missingness pattern available at prediction time matches the pattern seen during training; a model that learns from information unavailable in production will give an overly optimistic assessment of its performance.One simple way to incorporate this information is the missing indicator method (MIM), where missing values are imputed while an additional binary feature records whether the original value was absent. Where missingness is informative, the model can learn from both the value itself and the fact that it was missing. Van Ness et al. (2023) show that this approach can improve predictive performance when missingness is informative, although in high-dimensional settings large numbers of uninformative indicators can themselves lead to overfitting. The same idea is handled implicitly by some modern tree-based models: gradient-boosted decision trees, for example, can learn default directions for missing values (Chen and Guestrin, 2016), allowing the absence of a measurement to become part of the model's decision process rather than requiring it to be filled in beforehand. This illustrates the broader distinction between prediction and explanation: in a predictive system, a missing value may be a useful signal; in an inferential analysis, the same signal may represent precisely the source of bias that must be controlled.How we handle a missing value should therefore depend on why it is missing and what we are trying to learn from the data. There is no neutral decision; every approach carries assumptions about the process that produced the gaps. The best we can do is think carefully about that process and be explicit about what we do not know.···ReferencesChen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. https://doi.org/10.1145/2939672.2939785Cro, S., Morris, T. P., Kenward, M. G., & Carpenter, J. R. (2020). Sensitivity analysis for clinical trials with missing continuous outcome data using controlled multiple imputation: A practical guide. Statistics in Medicine, 39(21), 2815–2842. https://doi.org/10.1002/sim.8569Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161. https://doi.org/10.2307/1912352ICH E9(R1) Expert Working Group. (2019). Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials. International Council for Harmonisation. https://database.ich.org/sites/default/files/E9-R1_Step4_Guideline_2019_1203.pdfRobins, J. M., Rotnitzky, A., & Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89(427), 846–866. https://doi.org/10.2307/2290910Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581Safarov, R., Shomanova, Z., Nossenko, Y., Kopishev, E., Bexeitova, Z., & Atasoy, E. (2025). DynamicSeq2SeqXGB for PM₂.₅ imputation in extremely sparse environmental monitoring networks. PLoS One, 20(12), e0338788. https://doi.org/10.1371/journal.pone.0338788Sperrin, M., Martin, G. P., Sisk, R., & Peek, N. (2020). Missing data should be handled differently for prediction than for description or causal explanation. Journal of Clinical Epidemiology, 125, 183–187. https://doi.org/10.1016/j.jclinepi.2020.03.028Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009). Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ, 338, b2393. https://doi.org/10.1136/bmj.b2393Van Ness, M., Bosschieter, T. M., Halpin-Gregorio, R., & Udell, M. (2023). The missing indicator method: From low to high dimensions. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2447–2458. https://doi.org/10.1145/3580305.3599911Yan, X., Lee, S., & Li, N. (2009). Missing data handling methods in medical device clinical trials. Journal of Biopharmaceutical Statistics, 19(6), 1085–1098. https://doi.org/10.1080/10543400903243009