Development and Validation of Comorbidity Severity Adjustment Methods in Mortality Models for Acute Cerebrovascular Disease Using Survival and Machine Learning Analyses

TL;DR

This study improves mortality risk prediction for stroke patients by developing new comorbidity indices and using machine learning models.

Contribution

The study introduces recalibrated comorbidity indices (m-CCI and m-CCS) and demonstrates machine learning's effectiveness in mortality prediction.

Findings

The recalibrated m-CCI and m-CCS indices improved in-hospital mortality prediction for acute cerebrovascular disease patients.

Machine learning models like GBM and ANN achieved high predictive accuracy (AUCs of 0.835 and 0.830).

m-CCS consistently outperformed other comorbidity indices in mortality risk prediction.

Abstract

Background/Objectives: This study aimed to develop and validate comorbidity-based severity adjustment methods for acute cerebrovascular disease by recalibrating the Charlson Comorbidity Index (CCI) and constructing a CCS-based comorbidity index to improve mortality risk prediction. Methods: Using the Korea Disease Control Agency’s Discharge Injury In-depth Survey Dataset (2013–2022), we applied Cox proportional hazards regression and machine learning techniques, including LASSO, CART, Random Forests, GBM, and ANN, to recalibrate the CCI and develop a CCS-based comorbidity index. Results: The recalibrated Charlson Comorbidity Index (m-CCI) and the newly developed CCS-based comorbidity index (m-CCS) demonstrated improved predictive performance for in-hospital mortality. Among the machine learning models, GBM (AUC = 0.835) and ANN (AUC = 0.830) demonstrated the highest predictive accuracy, with m-CCS consistently outperforming other indices. Conclusions: The recalibrated m-CCI and newly developed m-CCS comorbidity indices enhance mortality risk adjustment for acute cerebrovascular disease patients in Korea. The superior performance of machine learning models underscores their potential for enhancing severity adjustment in hospital benchmarking and quality assessment.

Figures

Full text

1. Introduction

Acute cerebrovascular disease is a leading cause of mortality in South Korea, contributing significantly to the national disease burden [1]. Given its high fatality rate and long-term health implications, effective risk stratification and comprehensive hospital mortality management are crucial for enhancing patient survival and ensuring quality care. In-hospital mortality serves as a crucial metric for evaluating healthcare quality, necessitating rigorous adjustment for patient-level characteristics, particularly comorbidity burden, given its profound influence on patient outcomes [2]. While not the primary admitting diagnosis, comorbidities are critical determinants of patient severity and significantly impact mortality risk, thus requiring careful consideration in analytical models [3].

The Charlson Comorbidity Index (CCI) is a widely used tool for quantifying comorbidity burden by assessing the presence and severity of pre-existing medical conditions [4]. However, its applicability has been questioned over time due to changes in the prevalence and impact of comorbid conditions [5]. To address these concerns, Quan et al. proposed a modified version of the CCI based on administrative health data, which has been validated and implemented in various settings, including Korea’s Hospital Standardized Mortality Ratio (HSMR) calculation [6,7]. Studies have shown that this modified CCI demonstrates improved predictive accuracy for mortality compared to the original CCI, particularly in specific patient populations and healthcare contexts [8,9,10]. Despite its broad application, this model does not incorporate data from Korean hospitalized patients, limiting its ability to reflect Korea’s inpatient population characteristics. Moreover, the fixed weighting system applies the same weights across all primary diagnoses and only accounts for 17 comorbidities, potentially excluding other significant mortality-related conditions.

Unlike traditional comorbidity indices that are limited to a predefined set of conditions, the Clinical Classifications Software (CCS), version 2015.1, groups ICD-coded diagnoses into clinically meaningful categories, allowing for a more comprehensive classification of diseases [11]. This approach enhances severity adjustment by capturing a broader spectrum of comorbid conditions, particularly in complex patient populations, and improves hospital mortality risk standardization. Studies have demonstrated that CCS-based risk adjustment outperforms traditional indices like the Charlson Comorbidity Index (CCI) in mortality prediction. As a result, several national healthcare systems, including those in the United States and the United Kingdom, have adopted CCS-based models for hospital benchmarking and quality assessment [11,12].

Efforts to refine comorbidity adjustment for mortality prediction in the United Kingdom led to the development of a UK-specific comorbidity measure, known as the Summary Hospital-level Mortality Indicator (SHMI) [13]. This model, derived from UK national discharge data, was designed to more accurately reflect the characteristics of UK hospital discharges by incorporating updated comorbidity weightings. It is now widely utilized in the development of hospital mortality prediction models and in evaluating hospital performance with enhanced precision and clinical applicability. This adaptation involved revising the list of included comorbidities and recalibrating their weights based on more recent population data. Nevertheless, it has the limitation of not fully accounting for all comorbidities that influence mortality, highlighting the ongoing need for further refinement in severity adjustment models.

In our current study, we aimed to develop severity adjustment methods for patients with acute cerebrovascular disease using Korean hospital discharge data. Specifically, we utilized the Korea Disease Control Agency’s Discharge Injury In-depth Survey Dataset to construct a comorbidity-based severity adjustment model. By focusing on acute cerebrovascular disease, we aim to develop a tailored severity adjustment tool that more precisely captures its clinical characteristics. Additionally, this study seeks to recalibrate the Charlson Comorbidity Index (CCI) while also generating a CCS-based comorbidity index that considers a broader range of diseases. Finally, we validated these models to ensure their effectiveness in mortality risk adjustment for Korean inpatients.

2. Materials and Methods

2.1. Data Source and Study Population

This study utilized 1,803,611 records from the Korea National Hospital Discharge In-depth Injury Survey (KNHDIS), which is conducted annually by the Korea Disease Control and Prevention Agency (KDCA) and contains nationwide administrative data on hospitalized patients from 2013 to 2022.

The study sample was extracted following the methodology for developing severity-adjusted mortality models and the Hospital Standardized Mortality Ratio (HSMR) assessments in Korea. From the dataset, 38,497 inpatient records were identified in which the primary diagnosis fell under the Acute Cerebrovascular Disease category, as classified by the Clinical Classifications Software (CCS) diagnosis group 109. Table 1 presents the International Classification of Diseases, 10th Revision (ICD-10) codes included in CCS category 109: Acute Cerebrovascular Disease.

Patients transferred to another hospital, cases with unknown or unspecified discharge status (8157 cases), and same-day discharges (256 cases) were excluded. The final study included 30,084 cases, which were used to recalculate the Charlson Comorbidity Index (CCI) and construct the CCS-based comorbidity index.

To validate the comorbidity indices, we followed the methodology from previous studies [12,14] by splitting the dataset into a training set, comprising records from 2013, 2015, 2017, 2019, and 2021 (14,888 cases), and a validation set, consisting of records from 2014, 2016, 2018, 2020, and 2022 (15,196 cases).

2.2. Variables

The dependent variable in this study was in-hospital mortality (death or survival). The independent variables, considered in the recalibration and validation of the Charlson Comorbidity Index (CCI) and the CCS-based comorbidity index, included sex, age, insurance type, surgical status, emergency admission status, and comorbidities, aligning with the variables used in the calculation of Korea’s Hospital Standardized Mortality Ratio (HSMR).

With respect to comorbidities, three distinct approaches were employed to assess their impact: (1) the CCI, the Charlson Comorbidity Index (CCI) currently used in Korea; (2) the m-CCI, a recalibrated version of the CCI developed specifically for this study; and (3) the m-CCS, a modified comorbidity index constructed using the Clinical Classifications Software (CCS) from the Agency for Healthcare Research and Quality (AHRQ), USA.

2.3. Statistical Analysis

For recalibrating the Charlson Comorbidity Index (CCI) and developing the CCS-based comorbidity index, Cox proportional hazards regression was used to estimate new comorbidity weights using survival analysis. We tested the proportional hazard assumption using the Schoenfeld test. For the m-CCI, hazard ratios (HRs) and 95% confidence intervals (CIs) were estimated while adjusting for sex, age, insurance type, surgical status, emergency admission status, and 17 CCI-defined comorbidities. Similarly, for the m-CCS, HRs and 95% CIs were estimated with adjustments for sex, age, insurance type, surgical status, emergency admission status, and 260 CCS-defined comorbidities, and normalized in the same manner.

As a result, the final recalibrated m-CCI and m-CCS comorbidity weights were established, and three models were developed based on these indices:

Model 1: Adjusted for age, gender, type of insurance, surgical status, emergency admission status, and baseline CCI.

Model 2: Adjusted for age, gender, type of insurance, surgical status, emergency admission status, and modified CCI (m-CCI).

Model 3: Adjusted for age, gender, type of insurance, surgical status, emergency admission status, and modified CCS (m-CCS).

The validity of the Charlson Comorbidity Index (CCI), as well as the recalibrated m-CCI and m-CCS indices, was assessed using traditional logistic regression and various machine learning approaches. For machine learning analysis, this study adopted the methodology proposed by Desai et al., employing Least Absolute Shrinkage and Selection Operator (LASSO) regression, the Classification and Regression Tree (CART), Random Forests, and the Gradient-Boosted Model (GBM) [15]. Additionally, an Artificial Neural Network (ANN), a fundamental deep learning model, was incorporated to further validate the model performance.

Model performance was assessed using the area under the receiver operating characteristic curve (AUC) with 95% confidence intervals (CIs) in both the training and validation datasets. The AUC values were used to assess the discriminative ability of the models and to ensure robustness in performance between the training and validation phases, ensuring consistency and reliability in predictive performance across datasets.

All statistical analyses were performed using SAS 9.4 for Windows (SAS Institute, Cary, NC, USA) and Python 3.10.0 for Windows (Python Software Foundation, Wilmington, DE, USA).

3. Results

3.1. Baseline Characteristics

The study included a total of 30,084 cases, which were divided into training (n = 14,888) and validation (n = 15,196) datasets. Table 2 summarizes the baseline characteristics of the study population. The cohort consisted of 57.0% males and 43.0% females, with similar distributions in the training (56.4% male) and validation (57.7% male) sets. The mean age of the overall population was 65.9 ± 14.64 years, and the majority of patients were covered by National Health Insurance (93.1%), followed by Medical Aid (6.3%). Most patients (86.0%) did not undergo surgery, and 80.5% were admitted through emergency departments, with similar distributions across the training and validation groups.

3.2. Recalibration of the Charlson Comorbidity Index (CCI) and the Development of the CCS-Based Comorbidity Index Using Survival Analysis

The comorbidity weights for the Charlson Comorbidity Index (CCI) were recalibrated using Cox proportional hazards regression based on a total of 30,084 cases. We confirmed a relationship between time and residuals in proportional hazard assumption using the Schoenfeld test (p = 0.291). Comorbidities with statistically significant hazard ratios (p < 0.05) and HR ≥ 1 were normalized by dividing them by the lowest HR, leading to the final recalibrated modified CCI (m-CCI) weights.

Among the comorbidities, moderate or severe liver disease (HR = 2.880, p < 0.0001) had the highest recalibrated m-CCI weights, indicating a strong association with in-hospital mortality (Table 3).

Among the 260 CCS-defined comorbidities, 55 comorbidities with a p-value ≤ 0.05 and a hazard ratio (HR) ≥ 1 were used to calculate the m-CCS weights. Leukemia (HR = 7.224) had the highest weight, followed by pancreatic disorder (HR = 5.315) and cardiac arrest and ventricular fibrillation (HR = 5.144), showing strong associations with in-hospital mortality. The model incorporated sex, age, insurance type, surgical status, and emergency admission status to ensure comprehensive risk adjustment. We confirmed a relationship between time and residuals in the proportional hazard assumption using the Schoenfeld test (p = 0.214) (Table 4).

3.3. Distribution of CCI, m-CCI, and m-CCS Comorbidity Indices

The Charlson Comorbidity Index (CCI) has a mean value of 0.25, indicating that a large proportion of patients either have no comorbidities or very few, which aligns with a generally low comorbidity burden. The m-CCI, recalibrated based on hazard ratios, has a mean of 0.26, suggesting that the comorbidity burden is now more appropriately weighted and dispersed compared to the original CCI. The m-CCS has the highest mean value (0.55), indicating that it assigns higher overall comorbidity scores than both the CCI and m-CCI, reflecting a more comprehensive assessment of the comorbidity burden (Figure 1).

3.4. Validation of the Comorbidity Indices Using Machine Learning

Table 5 presents the area under the ROC curve (AUC) values and 95% confidence intervals (CIs) for different models validated on training (n = 14,888) and validation (n = 15,196) datasets. The m-CCS comorbidity index consistently demonstrated the highest predictive performance across all models, achieving the highest AUC values in both training and validation datasets. Among the models, GBM (AUC = 0.835 in training, 0.818 in validation) and ANN (AUC = 0.830 in training, 0.815 in validation) showed the highest overall AUC values, demonstrating their superiority over traditional regression models, while CART performed the weakest, with lower AUC values and less differentiation among the three indices. The m-CCI outperformed the original CCI, indicating that the recalibrated index enhances risk prediction accuracy (Figure 2).

4. Discussion

In this study, we developed severity adjustment methods for acute cerebrovascular disease patients using data from the Korea National Hospital Discharge In-depth Injury Survey (KNHDIS). By constructing a comorbidity-based severity adjustment model, we aimed to enhance the precision of risk adjustment for Korean inpatients. Our approach involved recalibrating the Charlson Comorbidity Index (CCI) and generating a CCS-based comorbidity index that incorporates a broader spectrum of diseases. Through rigorous validation, we demonstrated the effectiveness of these models in mortality risk adjustment, ensuring their applicability in assessing hospital performance and patient outcomes in the Korean healthcare setting.

Recent studies have underscored the necessity of refining comorbidity indices to enhance the accuracy of mortality risk adjustment [16,17]. The findings of this study have demonstrated the robust predictive validity of the recalibrated modified Charlson Comorbidity Index (m-CCI) and the newly developed comorbidity index based on Clinical Classifications Software (m-CCS). These indices are particularly effective in identifying high-risk conditions among patients with acute cerebrovascular disease, including moderate or severe renal disease, malignancies, leukemia, pancreatic disorders, and cardiac arrest. These results are particularly crucial for patients with acute cerebrovascular disease, for whom comorbidity burden plays a critical role in clinical outcomes.

From the distribution pattern of the comorbidity index, we revealed a more pronounced right-skewed distribution in the m-CCS model compared to CCI and m-CCI. This skewness indicates that while most patients have low comorbidity scores, a subset with severe comorbidities receives significantly higher scores. The greater skewness in the m-CCS model suggests its enhanced ability to distinguish patients with higher comorbidity burdens, assigning more precise and elevated scores. As a result, m-CCS captures a broader spectrum of disease severity, offering a more comprehensive assessment of comorbidity burden than CCI and m-CCI.

Furthermore, by utilizing the CCS-based comorbidity index (m-CCS), this study developed a tailored severity adjustment tool that accurately reflects the unique clinical characteristics of acute cerebrovascular disease, as demonstrated by its consistently highest AUC values. Unlike traditional indices, the m-CCS model allows for a more comprehensive evaluation of all disease categories, enabling a more refined risk adjustment approach. This advancement enhances the precision of severity assessment, thereby improving mortality risk prediction, hospital performance evaluation, and data-driven clinical decision-making in Korean healthcare settings.

By incorporating key demographic and clinical variables—including sex, age, insurance type, surgical status, and emergency admission status—this study contributes to a more refined risk adjustment methodology specifically for acute cerebrovascular disease. However, future research should focus on developing tailored severity adjustment tools that reflect the unique characteristics of various diseases, ensuring more precise risk stratification across different patient populations. Additionally, clearly identifying comorbidities that significantly impact mortality is crucial for guiding clinical decision-making and optimizing the management of high-risk patients. By providing clinicians with more accurate comorbidity-based risk assessments, these tools can support improved patient care and more effective interventions for critically ill individuals.

This study employed a combination of statistical, machine learning, and deep learning techniques to develop and validate severity adjustment methods. Given that severity-adjusted ratios are publicly disclosed at the national level, ensuring their accuracy and reliability through high-performance predictive models is crucial [14]. However, a key limitation of advanced analytical techniques, particularly deep learning, is their lack of interpretability, making it difficult to understand how specific outcomes are generated. Since clinicians play a vital role in patient care and mortality management, it is essential to provide transparent, evidence-based insights into the comorbidities that significantly impact mortality for each primary diagnosis. Enhancing the transparency and clinical applicability of risk adjustment models will enable healthcare professionals to integrate these tools into evidence-based practice, ultimately improving patient management and healthcare decision-making.

Despite the strengths of this study, several limitations should be acknowledged. First, this study was developed using discharge patient data exclusively from Korea to establish a Korean-specific severity adjustment method. Consequently, the results may reflect characteristics unique to the Korean healthcare system and may not be directly applicable to other populations. In addition, as the dataset lacked hospital-level and socioeconomic variables, future research should incorporate these factors to enhance the model’s generalizability and applicability across different settings. Second, this study used an alternating-year split method based on prior research, but this approach may introduce temporal bias due to changes in clinical practices and patient characteristics over time. Moreover, k-fold cross-validation or resampling methods were not used to assess generalizability or reduce overfitting, which we acknowledge as a limitation for future improvement. The severity adjustment tool was developed using statistical analysis, which, while robust, does not fully incorporate clinical expertise from physicians. Integrating clinical insights is necessary to enhance its practical utility. Additionally, due to sample size constraints, more advanced artificial intelligence techniques, such as deep learning, could not be fully applied in this study. Future research should address these limitations by expanding data availability, incorporating a broader range of AI methodologies, and adopting approaches that reflect the unique characteristics of each country to refine severity adjustment models. Such improvements will contribute to more precise risk stratification, better resource allocation, and an overall enhancement in healthcare system efficiency.