Applying machine learning to identify unrecognized COVID-19 deaths recorded as other causes of death in the United States

Prior efforts to quantify unrecognized COVID-19 deaths

Research on unrecognized COVID-19 deaths in the US has largely relied on excess mortality models, which estimate mortality attributable to the pandemic by comparing observed all-cause deaths to those expected on the basis of prepandemic trends (27). Prior studies (with different modeling specifications) have estimated that excess mortality exceeded reported COVID-19 mortality by 28% in early 2020 (28), 38% in 2020 (29), and 14% in 2020 (30).

Excess mortality models have largely relied on all-cause mortality data, which include deaths from external causes (e.g., intentional or unintentional injuries) unlikely to be caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection in the short term (31, 32). A prior study excluded external-cause deaths to estimate natural-cause excess deaths, which exceeded reported COVID-19 deaths by 16% from March 2020 to August 2022 (33). This study and a prior study from California showed strong temporal concordance between reported COVID-19 deaths and excess deaths reported to other natural causes, suggesting that many such deaths were unrecognized COVID-19 deaths (33, 34).

As more detailed mortality data have become available, researchers have sought to produce more refined estimates of excess mortality (35, 36). One prior study used age-, state-, and cause-specific excess mortality models to conclude that 84% of pandemic-era all-cause excess mortality could be directly attributed to SARS-CoV-2 infection (37). Excess mortality models have value in the context of emerging health crises as a means of estimating total mortality attributable to the crisis and, in the case of the pandemic, approximating unrecognized COVID-19 deaths (8, 27). This prior evidence, however, is suggestive but not conclusive because excess mortality models do not directly distinguish between deaths from SARS-CoV-2 infection and those caused by pandemic-era health care interruptions and/or social and economic impacts among individuals with no recent SARS-CoV-2 infection (38–40). Differentiating deaths due to infection from deaths due to pandemic response policies is at the heart of the policy debate about appropriate emergency response, so this uncertainty is a critical limitation (41).

Potential inequities in unrecognized COVID-19 deaths

Although a large body of research has examined geographic and sociodemographic inequities in reported COVID-19 and excess mortality during the pandemic (42–50), few studies have explored inequities in COVID-19 deaths that have gone unrecognized. One prior study found that excess deaths attributed to non–COVID-19 causes were more common in counties with lower socioeconomic status, greater prevalence of preexisting health conditions, and a greater fraction of non-Hispanic Black residents (51). Another study found that excess deaths attributed to non–COVID-19 natural causes were more common in rural areas, the South, and the West (33). While these studies did not examine unrecognized COVID-19 deaths directly, they suggest that the death investigation system may have performed unevenly during the pandemic (8).

Applying machine learning to estimate unrecognized COVID-19 deaths

Machine learning algorithms used to classify the cause of death may outperform excess death models if there is a reliable set of correctly classified deaths available to train the algorithm. Wrigley-Field et al. (52) proposed using cause and place of death information to differentiate deaths directly and indirectly related to COVID-19. The logic behind this approach is that although individuals who died outside a hospital were often not tested for SARS-CoV-2 infection, such testing was nearly universal among hospital patients, starting as early as April 2020 (53–56). Furthermore, while there were many excess natural-cause deaths attributable to the pandemic that were reported to causes other than COVID-19 in out-of-hospital settings, the number of reported COVID-19 deaths in hospitals was similar to the number of excess natural-cause deaths, suggesting that COVID-19 deaths were accurately identified here (34). Thus, in-hospital deaths, during this period of near-universal testing, provide a pool of high-quality training data for classification of whether a death was due to SARS-CoV-2 infection for use in predicting whether deaths in out-of-hospital settings were likely to be COVID-19 related. Here, we applied predictive machine learning to this strategy, building on prior research on machine learning for predictive inference to improve the accuracy of disease diagnosis and postmortem death investigation (57–61).

This study had two research questions: (i) How many unrecognized COVID-19 deaths occurred from March 2020 to December 2021 in the US, and (ii) did unrecognized COVID-19 deaths occur in some sociodemographic populations disproportionately? We quantified unrecognized COVID-19 deaths using two distinct but related metrics: (i) the predicted number of unrecognized COVID-19 deaths in out-of-hospital settings (i.e., the difference between the number of out-of-hospital deaths predicted by our machine learning algorithm as likely to be COVID-19 related and the officially reported number of out-of-hospital COVID-19 deaths) and (ii) an adjusted reporting ratio (ARR) that estimated underreporting of COVID-19 deaths across all settings of death by weighting the estimated degree of misclassification of COVID-19 deaths in out-of-hospital settings by the proportion of deaths that occurred in out-of-hospital settings. The ARR metric aligns with our research question of examining whether unrecognized COVID-19 deaths occurred in some sociodemographic populations disproportionately (across all settings of death).

Finding 1: National estimate of unrecognized COVID-19 deaths

From March 2020 to December 2021, our estimate of total COVID-19 deaths in the US {995,787 deaths [95% uncertainty interval (UI): 990,313 to 1,001,363]} exceeded the 840,251 COVID-19 deaths officially reported (i.e., those with COVID-19 listed anywhere on their death certificate) by 19% [ARR: 1.19 (95% UI: 1.18 to 1.19)]. This equals 155,536 predicted unrecognized COVID-19 deaths (95% UI: 150,062 to 161,112) in out-of-hospital settings during this period.

The number of COVID-19 deaths in decedents’ homes was 160% higher than officially reported [ARR: 2.60 (95% UI: 2.56 to 2.65)], indicating that there were 111,245 predicted unrecognized COVID-19 deaths that occurred in homes (95% UI: 108,372 to 114,210) (Table 1). Predicted unrecognized COVID-19 deaths also occurred in hospice settings [17,346 deaths (95% UI: 16,832 to 17,873)], outpatient and emergency room settings [14,832 deaths (95% UI: 14,364 to 15,315)], and other settings not elsewhere classified [9452 deaths (95% UI: 9011 to 9926)].

We found similar model performance across racial and ethnic groups, sex, and educational attainment (table S1 and fig. S1), the rural-urban continuum (table S2), and states (table S3). Sensitivity analyses demonstrated that our national estimate remained consistent even when we made a different model selection {ARR: 1.17 [95% confidence interval (CI): 1.16 to 1.18]} (table S4). We also found that our results were conservative compared to other (empirically untestable) assumptions of misclassification (ARR increased up to a maximum of 1.44).

Finding 2: Geographic and temporal differences in unrecognized deaths

COVID-19 deaths were most likely to go unrecognized in the South (Table 1). The estimated number of COVID-19 deaths was 31% higher than officially reported (95% UI: 29 to 33%) in the West South Central division, 26% higher than officially reported (95% UI: 24 to 27%) in the Middle Atlantic division, and 25% higher than officially reported (95% UI: 23 to 28%) in the East South Central division.

Figure 1 represents the degree of predicted underreporting of COVID-19 deaths in the US by county and state. The states that we identified where COVID-19 deaths were most likely to go unrecognized were Alabama [ARR: 1.67 (95% UI: 1.61 to 1.74)], Oklahoma [ARR: 1.51 (95% UI: 1.46 to 1.57)], and South Carolina [ARR: 1.47 (95% UI: 1.42 to 1.52)]. The states with the largest estimates of predicted unrecognized COVID-19 deaths were Texas [24,024 deaths (95% UI: 22,381 to 25,740)], New York [23,005 deaths (95% UI: 21,690 to 24,330)], California [11,613 deaths (95% UI: 10,408 to 12,846)], Alabama [11,501 deaths (95% UI: 10,435 to 12,583)], Florida [7718 deaths (95% UI: 6848 to 8592)], and South Carolina [7224 deaths (95% UI: 6468 to 8027)] (table S5).

Figure 2 shows the temporal evolution of COVID-19 mortality reporting by pandemic wave and month. COVID-19 deaths were most likely to go unrecognized in the initial wave of the pandemic [March to May 2020; highest ARR during this wave: 1.49 (95% UI: 1.48 to 1.51)]. Predicted unrecognized COVID-19 deaths declined in likelihood but continued to occur in subsequent waves [highest monthly ARR during the second wave: 1.19 (95% UI: 1.17 to 1.21); during the Alpha wave: 1.34 (95% UI: 1.32 to 1.36); during the Delta wave: 1.23 (95% UI: 1.22 to 1.25); during the start of the Omicron wave: 1.09 (95% UI: 1.07 to 1.11)]. By month, the largest absolute number of predicted unrecognized COVID-19 deaths occurred in the months of January 2021 [35,665 deaths (95% UI: 33,248 to 38,208)] and April 2020 [32,110 deaths (95% UI: 30,978 to 33,256)] (table S6). As shown in the bottom panel of Fig. 2, the peaks in predicted unrecognized COVID-19 deaths correspond to the monthly peaks in official COVID-19 deaths: The largest number of official COVID-19 deaths occurred in January 2021 along with another peak in April 2020. This correspondence supports the finding that the unrecognized COVID-19 deaths we predicted were COVID-19 deaths.

Finding 3: Differences in unrecognized deaths by decedent characteristics

We next examined the predicted accuracy of COVID-19 reporting by decedent characteristics as recorded on death certificates (Fig. 3). For age, COVID-19 deaths between the ages of 65 to 84 years were most likely to go unrecognized [ARR for 65 to 74 years: 1.21 (95% UI: 1.21 to 1.22); ARR for 75 to 84 years: 1.22 (95% UI: 1.21 to 1.23)]. COVID-19 deaths recorded as male were more likely to go unrecognized than deaths recorded as female [ARR for male: 1.22 (95% UI: 1.21 to 1.23); ARR for female: 1.14 (95% UI: 1.13 to 1.15)]. COVID-19 deaths were also more likely to go unrecognized for those identified as having less than any high school education compared to those recorded as having more education [ARR for less than high school: 1.29 (95% UI: 1.28 to 1.31); ARR for high school diploma or equivalent: 1.18 (95% UI: 1.17 to 1.18); ARR for some college: 1.15 (95% UI: 1.14 to 1.16)]. There was also variability by marital status.

COVID-19 deaths among individuals identified as Hispanic were most likely to go unrecognized [ARR: 1.31 (95% UI: 1.30 to 1.32)]. Compared to decedents recorded as non-Hispanic white [ARR: 1.15 (95% UI: 1.14 to 1.16)], deaths were also more likely to go unrecognized if the decedent was listed on certificates as non-Hispanic American Indian and Alaska Native [ARR: 1.24 (95% UI: 1.21 to 1.28)], non-Hispanic Asian [ARR excluding Indian: 1.24 (95% UI: 1.22 to 1.27); ARR for Indian: 1.23 (95% UI: 1.19 to 1.26)], or non-Hispanic Black [ARR: 1.19 (95% UI: 1.18 to 1.20)]. In absolute terms, 78,561 predicted unrecognized COVID-19 deaths (95% UI: 73,783 to 83,447) occurred among non-Hispanic white adults, 43,132 deaths (95% UI: 41,632 to 44,622) among Hispanic adults, 23,617 deaths (95% UI: 22,269 to 24,985) among non-Hispanic Black adults, 5772 deaths (95% UI: 5285 to 6328) among non-Hispanic Asian adults, and 2318 deaths (95% UI: 1982 to 2650) among non-Hispanic American Indian and Alaska Native adults (table S7). The unrecognized COVID-19 deaths that our machine learning approach predicted were attributed to a variety of underlying causes of death including Alzheimer’s disease and related dementia, cardiovascular disease, and diabetes (table S8).

Finding 4: Differences in unrecognized deaths by county characteristics

Our predictions showed that reporting of COVID-19 deaths also varied by county-level characteristics (Fig. 4). COVID-19 deaths were more likely to go unrecognized in counties in the lowest quintile of median household income [ARR: 1.34 (95% UI: 1.31 to 1.36)] and the lowest quintile of the proportion of residents with some or more college [ARR: 1.37 (95% UI: 1.33 to 1.41)]. COVID-19 deaths were more likely to go unrecognized in counties with the most residents reporting poor or fair health [ARR: 1.30 (95% UI: 1.28 to 1.33)] and with a higher percentage of residents living with diabetes [ARR: 1.33 (95% UI: 1.30 to 1.35)]. COVID-19 death reporting accuracy also varied across the rural-urban continuum.

Implications for the COVID-19 pandemic

Our machine learning approach predicted 995,787 COVID-19 deaths in the US from March 2020 to December 2021 and 155,536 unrecognized COVID-19 deaths. This finding adds to the evidence base documenting uncounted COVID-19 deaths nationally (33, 34, 37). Prior attempts to study unrecognized COVID-19 deaths have largely relied on excess mortality models, which cannot directly distinguish whether an excess death was due to COVID-19 or the pandemic’s indirect effects (41). Our national estimate of total predicted COVID-19 deaths is similar to the estimate of the US Centers for Disease Control and Prevention (CDC) of just under 1 million excess deaths for this period (62). Excess mortality models assume that projected historical mortality trends accurately estimate expected mortality in the absence of the pandemic and are sensitive to the choice of historical period (27). Our machine learning approach relies on different assumptions—(i) that there was no measurement error for COVID-19 deaths in hospitals and (ii) that a model trained on in-hospital deaths is transportable to out-of-hospital settings. It also enables subgroup estimates that are typically infeasible with excess mortality models. Estimates in this study should be interpreted alongside other estimates derived using different methodologies to support evidence triangulation (63).

The primary contribution of this study is our predictions showing that unrecognized COVID-19 deaths occurred unevenly, disproportionately affecting decedents identified as Hispanic, American Indian, Alaska Native, Black, and Asian; decedents with less than high school education; and decedents who resided in counties with lower socioeconomic status and worse prepandemic health. Together, this study indicates that the US death investigation system reported COVID-19 deaths in a systematically inequitable way that hid the true extent of pandemic mortality and inequities. While this study does not isolate mechanisms in the death investigation system, the communities affected by the undercounting of COVID-19 deaths could be interpreted as a pattern of structural racism, classism, and ableism in the death investigation system that warrants further research and policy attention (22–26).

Some public health groups and community organizations have expressed concerns that incomplete reporting of COVID-19 deaths contributed to an insufficient pandemic response (5, 25, 40, 64–66). Many of the communities in which we predicted unrecognized COVID-19 deaths were also the populations where the largest number of total COVID-19 deaths occurred (e.g., Latino populations, Indigenous populations, and people with preexisting health conditions or disabilities), a consequence of health and social policy failures (67). In this way, whether intentional or not, undercounting of COVID-19 deaths has functioned to absolve politicians and health policy-makers of responsibility for some of these failures.

This study did not directly examine actors, policies, and structures in death investigation and other health systems that may have contributed to inaccuracies in COVID-19 death reporting. Death investigators faced difficult circumstances during the pandemic, and many put themselves at occupational health risk to investigate COVID-19 deaths (14). This study’s predictions are suggestive of biased outcomes in the investigations carried out in this challenging context, a possibility that the field’s professional ethics recognize as a risk to guard against (68). Some policies and structural factors that may be relevant include inadequate and inequitable funding for death investigation offices to hire staff and conduct investigations (e.g., postmortem COVID-19 testing) (69, 70), variability in training required for death investigators (14, 16, 71), and elected roles presenting conflicts of interests (i.e., sheriff-coroners) (11, 19). While community variables (i.e., access to COVID-19 testing, health system distrust, and preexisting conditions that may complicate death investigation) could all also influence reporting accuracy, a well-functioning death investigation system would ideally account for these determinants (12, 13, 17, 18). Another mechanism that may have contributed to inaccuracies in reporting is COVID-19 stigma and related political attitudes (72, 73). These factors may have affected whether a decedent sought testing in the weeks before their death, whether their family members reported COVID-19 symptoms to an investigator, and whether investigators followed established procedures (12, 14, 74–76).

Broader implications for death investigation systems

The machine learning approach used in this study could be further developed and adapted for use in public health responses in a number of settings where cause-of-death investigation is incomplete, delayed, and/or suspected to be biased (77, 78). In the case of COVID-19, numerous policies and programs (e.g., Federal Emergency Management Agency funeral assistance program) were designed around officially reported COVID-19 deaths (4, 8, 79). Other settings that may be particularly relevant for the application of these methods are deaths related to drug overdose (80), deaths in police custody (15), deaths related to Alzheimer’s disease and related dementias (81), and other public health emergencies like extreme heat–related deaths (20, 82). While machine learning approaches may have utility for improving estimates using existing data, these tools should not come at the expense of larger investments in death investigation system reform, efforts to study and address bias in death investigation systems, and programs to build death investigation infrastructure globally (1, 10, 11, 77).

Study design

Our machine learning approach consisted of four general methodological steps:

1) We tuned and selected a model trained on inpatient COVID-19 deaths using nested cross-validation.

2) We used this model to predict COVID-19 deaths in out-of-hospital settings (all other settings of death).

3) We evaluated differences in unrecognized COVID-19 deaths across sociodemographic populations. We estimated (i) the predicted number of unrecognized COVID-19 in out-of-hospital settings and (ii) the ARR.

4) We conducted additional analyses for model fairness and model misclassification.

We describe these steps below and provide additional detail in the Supplementary Text. This study did not involve human participants and was deemed exempt from review by the Stanford University Institutional Review Board.

Assumptions underlying our approach

Our approach to identifying unrecognized COVID-19 deaths involved training a machine learning algorithm on inpatient COVID-19 deaths to predict COVID-19 deaths in other settings (60, 61). Two core assumptions underlie this approach. First, the approach assumes that there was no measurement error in COVID-19 cause-of-death assignment in inpatient hospitals. This assumption is supported by the widespread (and often mandatory and universal) COVID-19 testing of patients in these settings in 2020 and 2021 (53–56), evidence that excess natural-cause deaths have equaled reported COVID-19 deaths in hospital settings (34), and research showing that clinicians in hospital settings adhered to national reporting guidance more closely than other death certifiers (2, 14, 19, 83). While it is possible that this assumption was partially violated in the early months of the pandemic, less than one-third of COVID-19 deaths occurred out of hospital during this period (84), and potential bias would result in an underestimation of unrecognized COVID-19 deaths, not an overestimation. Second, our approach assumes that the model is transportable from inpatient hospital to out-of-hospital settings. This includes assuming that predictors of COVID-19 deaths (i) were recorded with the same level of accuracy (i.e., that sociodemographic variables were either reported accurately or that any misreporting occurred similarly across settings of death), (ii) follow the same distribution, and (iii) have the same strengths of association in inpatient hospital versus out-of-hospital settings. This assumption is empirically untestable and represents the largest source of potential bias in our approach. However, it gains credence from the specific predictors that carry the most weight in our models, whose accuracy and predictive power are unlikely to vary substantially by place of death. We also tested the effect of differences in the distribution of predictors between inpatient hospital and out-of-hospital settings on the performance of our machine learning model. We provide a more detailed discussion of these assumptions in the “Rationale for our approach” section of the Supplementary Text.

Data

We used restricted-use individual-level multiple cause-of-death files from the National Center for Health Statistics to predict total COVID-19 mortality and estimate unrecognized COVID-19 deaths. There were 6.86 million recorded US deaths in 2020 and 2021. Our sample included adults 25 years and older dying from natural causes from March 2020 through December 2021. Deaths from external causes were excluded because we hypothesized that deaths from SARS-CoV-2 infection would be unlikely to be attributed to external causes of death (e.g., drug poisoning and homicide). There were 1,878,456 death certificates from adults 25 years or older dying from natural causes in inpatient hospital settings. This was the “gold standard training data” where we assumed no measurement error in COVID-19 deaths and which we used to build our model. There were 3,850,161 death certificates from adults 25 years or older dying from natural causes across all other settings of death. This was the “out-of-hospital prediction data” where we sought to predict whether deaths were likely to be COVID-19 related. We defined officially reported COVID-19 deaths as death certificates with ICD-10 code U07.1 listed as an underlying or contributing cause of death, consistent with the CDC definition (85).

For our machine learning model, we used information on causes of death (i.e., underlying and up to 20 contributing), decedent characteristics (i.e., age, sex, race, ethnicity, educational attainment, marital status, and smoking status as recorded on the death certificate, which has important limitations given that it is typically sourced from other records or surviving relatives as described in the Supplementary Text), temporal information (i.e., pandemic month of death and seasonality), geography (i.e., county and state of death and residence), and place of death. The eight places of death were hospital inpatient, hospital outpatient or emergency department, decedent’s home, nursing home or long-term care facility, hospice, dead on arrival to a medical facility, other, or unknown (86). We also linked decedents to several county of residence characteristics: (i) rurality, (ii) percentage who were homeowners, (iii) median household income, (iv) percentage who completed some college, (v) percentage with poor or fair health, and (vi) percentage with diabetes. We hypothesized that these factors may have influenced reporting accuracy by affecting access to COVID-19 testing, resources available to death investigators (i.e., ability to undertake postmortem testing), and case complexity (i.e., likelihood of misassignment to comorbidities) (2, 17, 87–89). We also linked decedents to two time-varying county factors: (i) monthly vaccination rates and (ii) community COVID-19 transmission levels.

Model selection and tuning

We compared four modeling approaches and four covariate sets for a total of 16 candidate models. We considered logistic regressions with elastic net regularization as our reference model, along with three tree-based alternatives that could offer potential improvements. These included random forests (90), extreme gradient boosting trees (XGBoost) (91), and light gradient boosting machine models (LightGBM) (92). The four covariate sets we considered were as follows: a “naïve” covariate set with all predictors along with county and contributing cause of death fixed effects, a “county fixed effects only” covariate set with all predictors along with county fixed effects, a “contributing causes fixed effects only” covariate set with all predictors along with contributing cause of death fixed effects, and a “no fixed effects” covariate set with all predictors and no fixed effects.

We assessed the performance of each of the 16 candidate models in classifying COVID-19 deaths within the pool of inpatient deaths (for which we assumed that there was no measurement error for officially reported COVID-19 deaths) using a nested cross-validation approach with a 60-20-20 split. Specifically, we used 60% of the inpatient deaths for model fitting, 20% for validation and hyperparameter tuning (i.e., Bayesian optimization as detailed in the Supplementary Text), and 20% for testing out-of-sample model performance. We executed this procedure a total of five times across mutually exclusive 20% test sets and calculated the out-of-sample performance as the average across the five test sets.

We selected the XGBoost model with the “no fixed effects” covariate set as our primary prediction model on the basis of it having the highest area under the receiver operator curve (ROC AUC: 0.90; sensitivity: 0.79; specificity: 0.85). We also selected LightGBM with the “no fixed effects” covariate set as a sensitivity analysis on the basis of its high performance.

Prediction of unrecognized COVID-19 deaths

We refit the selected primary model using all inpatient data and then used the model to predict COVID-19 deaths in out-of-hospital settings (all other settings of death). We estimated the absolute number of predicted unrecognized COVID-19 deaths in a population of interest by comparing the number of predicted COVID-19 deaths in out-of-hospital settings for a population to the officially reported number of COVID-19 deaths in out-of-hospital settings for that population. We estimated model-based uncertainty by using 5000 bootstrapped samples of the out-of-hospital predictions and reporting the 2.5th and 97.5th quantiles. Our predictions are based on the median of 5000 bootstrapped samples.

We also calculated the ARR, which was quantified as total predicted COVID-19 deaths (the sum of official in-hospital COVID-19 deaths and predicted out-of-hospital COVID-19 deaths) divided by official COVID-19 deaths across all settings of death. The ARR allows for a comparison of potential underreporting of COVID-19 deaths across all settings of death and accounts for differences in the percentage of deaths in out-of-hospital settings across populations.

An ARR > 1 indicates that our machine learning approach predicted more COVID-19 deaths in a population than the number of officially reported COVID-19 deaths. This suggests that COVID-19 deaths were underreported in official COVID-19 death counts for that population. An ARR < 1 indicates that COVID-19 deaths were overreported relative to the machine learning predictions. Populations and geographic areas with an ARR < 1 were infrequent. One reason that a population or geographic area could have an ARR < 1 is if death certifiers recorded people as dying from COVID-19 when they had COVID-19 but actually died from another unrelated cause. Especially early in the pandemic, states and death certifiers had different protocols that influenced how conservative they were with assigning COVID-19 deaths. For example, while most states defined COVID-19 deaths as deaths occurring within 30 days of a COVID-19 diagnosis, Massachusetts included deaths occurring within 60 days until March 2022 (26).

Model fairness and misclassification

We assessed model fairness across race and ethnicity, sex, educational attainment, the rural-urban continuum, and states by calculating the ROC AUC parity, false negative rate parity, and false positive rate parity in the inpatient data. Specifically, we used fivefold cross-validation. This involved splitting inpatient data into five subsets and using four of the subsets to predict the fifth subset. This procedure was conducted five times, and results were averaged. To confirm that the fairness metrics we selected for this study were not masking absolute differences in predictive performance, we also constructed subgroup-specific precision-recall curves for racial groups.

Given that we do not know the true COVID-19 status for any subset of out-of-hospital deaths, it was not possible to estimate model misclassification metrics for the model’s performance among out-of-hospital deaths. Instead, we estimated the sensitivity of our results to model misclassification on the basis of model performance for inpatient deaths. We adapted a prior approach (93) that involved estimating the false discovery rate and false omission rate among the inpatient deaths and applying these metrics to the out-of-hospital predictions by drawing from a binomial distribution and switching the outcome. Further details about this sensitivity analysis for model misclassification are provided in the Supplementary Text.

Funding:

The content is solely the responsibility of the authors and does not necessarily represent the official views of the study sponsors. We acknowledge receiving financial support from the W.K. Kellogg Foundation (to A.C.S.) and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (P2C HD041023 to E.W.-F.).

Author contributions:

M.V.K. and A.C.S. conceptualized the study, acquired the data, supervised the study, and acquired funding for the research. M.V.K. led the data analysis, with contributions from Z.R.L., R.V.R., and B.H. All authors contributed to interpretation of the data and writing of the manuscript.

Competing interests:

The authors declare that they have no competing interests.

Data, code, and materials availability:

This PDF file includes: