Ischemic stroke is a leading cause of mortality, long-term disability, and high healthcare costs in the US. In light of this clinical and financial burden, the Centers for Medicare & Medicaid Services (CMS) has decided to incorporate ischemic stroke measures of 30-day mortality and hospital readmission into its current pay-for-performance program. This decision has come under intense scrutiny, as many clinicians and researchers believe that the current risk adjustment model is inadequate because it does not include a measure of stroke severity. Due to its well-documented importance in individual-level prediction, there is concern that excluding a measure of stroke severity from risk adjustment will lead to incorrect rankings of hospital performance, i.e. hospital profiling. However, administrative datasets used in CMS currently do not capture a measure of stroke severity, such as the National Institutes of Health Stroke Scale (NIHSS), and in clinical databases which capture NIHSS, it is frequently missing. Little work has been done to asses if the documentation of NIHSS is biased, and if so, what impact bias would have on hospital-level estimates of mortality. In this study, we analyzed data from ischemic stroke patients from an existing stroke registry to identify patterns and characteristics that predict NIHSS documentation at the patient- and hospital-level. Next, we tested for the presence of selection bias in patients with documented NIHSS using the Heckman Selection Model. Finally, using computer simulations, we estimated the impact of missing NIHSS data on hospital profiling of 30-day mortality, under different assumptions about the prevalence and mechanism of missing NIHSS data. We found that patients with documented NIHSS were, in fact, a biased subsample of all ischemic stroke patients. Documentation of NIHSS was driven by a combination of patient-level and hospital-level factors. At the patient- and hospital-level, analyses suggested that patients with more severe strokes (i.e. increased NIHSS score) were better documented than patients with less severe strokes. These findings were confirmed using the Heckman Selection Model. However, in both analyses, we found that the amount of bias was modest. In computer simulations, we quantified the impact that missing data would have on the accuracy of hospital ischemic stroke profiling, under different assumptions about how NIHSS data was missing. Any effect of missing NIHSS mechanism was trumped by the impact of missingness on sample size. Because patients with missing NIHSS data were dropped from risk-adjustment models as documentation of NIHSS decreased, the accuracy of hospital risk-standardized mortality rates (RSMRs) estimated by the hierarchical logistic model deteriorated. All of our findings were substantially modified by the hospital ischemic stroke volume, with low volume hospital suffering the worst accuracy. These results are a reflection of the fact that the loss of sample size (either through the documentation rate or hospital volume), increases the amount of shrinkage in RSMR estimates, which makes any random noise more impactful on changes in RSMR. Overall, our findings raise concerns about the addition of NIHSS data into risk adjustment models for hospital-level ischemic stroke outcomes, and illustrate shortcomings in current methodologies used to profile hospitals. It is crucial that data used in risk adjustment for hospital profiling be documented with very high levels of completeness.
Read
