Comparing High-Flow Nasal Cannula and Non-Invasive Ventilation in Critical Care: Insights from Deep Counterfactual Inference

TL;DR

This study uses a deep learning model to compare high-flow nasal cannula and non-invasive ventilation in ICU patients, showing better outcomes when treatment aligns with model predictions.

Contribution

The novel contribution is a deep counterfactual model integrating representation learning and normalizing flows to estimate individual treatment effects in critical care.

Findings

Concordant treatment at UCSD reduced odds of invasive ventilation and mortality.

At UCI, concordant treatment was strongly linked to improved mortality outcomes.

Model-guided strategies improve outcomes for critically ill patients.

Abstract

Randomized trials comparing high-flow nasal cannula (HFNC) and non-invasive positive pressure ventilation (NIV) for acute respiratory failure (ARF) offer population-level guidance but often fail to capture individual variability in treatment response. In this retrospective study, we identified intensive care units (ICU) patients at risk of invasive mechanical ventilation (IMV) using a previously published risk prediction model. Patients who first received HFNC or NIV after crossing the high-risk threshold formed the early treatment cohort. We developed a deep counterfactual model that integrates representation learning, conditional normalizing flows, and confounder adjustment to estimate individualized treatment effects (ITEs) between HFNC and NIV. Treatment concordance, defined as alignment between the model’s recommendation and the treatment actually administered, was assessed using multivariate logistic regression. At UC San Diego Health (UCSD), concordant treatment was associated with significantly reduced odds of IMV (odds ratio [OR] = 0.661 for NIV; 0.677 for HFNC) and mortality or hospice discharge (OR = 0.679 for NIV; 0.749 for HFNC). At UC Irvine Health (UCI), concordant treatment was also linked to improved outcomes, particularly for mortality or hospice discharge (OR = 0.092 for NIV; 0.088 for HFNC). These findings highlight the value of individualized, model-guided respiratory support strategies in improving outcomes for critically ill patients.

Figures

a](#F1)), the most influential features included severity scores (e.g., coaSOFA, HRSIRS), pre-hospital factors (e.g., preLOS), and clinical interventions such as anesthesia and pain medication use, suggesting that patient acuity and treatment context strongly influenced individualized treatment recommendations. At the UCI site ([Figure 1b](#F1)), laboratory values and organ-specific dysfunction, particularly Red_cell_width_delta, renalSOFA, and Sodium, were more dominant, indicating site-specific variation in model interpretation. While some features such as coaSOFA and Phosphate_delta appeare

Full text

Introduction

Acute respiratory failure (ARF) is a common reason for hospitalization, both in general wards and intensive care units (ICU). Oxygen therapy, generally with simple nasal cannula, is often the initial treatment modality for those patients with hypoxemia. If respiratory failure progresses, and patients require more than 6–8L/min of oxygen to maintain adequate oxygen saturation or exhibit respiratory distress or hypercarbia, treating physicians escalate respiratory support in several different way including high flow nasal cannula (HFNC; airflows up to 50–60L/min), non-invasive positive pressure ventilation (NIV), or invasive mechanical ventilation (IMV). While the choice of the initial treatment modality is important and could influence patient outcomes, there remains clinical equipoise surrounding this treatment decision as several randomized controlled trials have shown mixed results.[2–11]

Medical society guidelines provide general guidance and advice for some specific clinical contexts. For example, the European Respiratory Society (ERS) and American Thoracic Society (ATS) guidelines[12] support NIV use for acute exacerbation of chronic obstructive pulmonary disease (AECOPD), cardiogenic pulmonary edema, immunocompromised patients with ARF, post-operative respiratory failure, palliative treatment of dyspnea, chest trauma patients with ARF, and preventing post-extubation respiratory failure in high-risk patients. However, the guidelines do not make recommendations about asthma, de novo ARF, or pandemic viral illness. Guideline updates from the ERS in 2022[13] added that patients with acute hypoxemia respiratory failure should generally be treated with HFNC over NIV, HFNC or NIV is equivocal for post-operative patients at high-risk of respiratory complications, and removed the recommendation for NIV use in immunocompromised patients. Many of these recommendations are conditional with low certainty of evidence. Real-world clinical care is often complicated, and patients can have multiple co-occurring conditions with varying indications or contraindications (e.g., multifocal pneumonia, pulmonary edema and AECOPD, neuromuscular weakness, facial trauma or surgery, large beards, inability to manage secretions or protect airway, severe encephalopathy, hemodynamic instability, gastric or esophageal pathology, etc.) for either HFNC or NIV.

Another feature complicating decisions regarding oxygen delivery modality is that ICU patients themselves are often inhomogeneous and many of the conditions commonly encountered in the ICU (sepsis, ARF, etc.) are syndromes composed of a variety of pathophysiologic states and severities. Controlling a myriad of disease states and patient-specific differences in randomized clinical trials in this population is difficult. It is therefore unsurprising that trials in this arena commonly fail to demonstrate a difference between treatment and control groups, show mixed results among studies, or fail to provide generalizable results. One explanation for these mixed findings is that the average treatment effect (ATE) captured by randomized control trials (RCTs) may hide variations of the treatment’s effect on a clinical outcome across levels of patient characteristics. Another way to rephrase is that even with a null overall result, due to heterogeneity of treatment effect (HTE), some patients may benefit from an intervention, others are unchanged, and others may even be hurt.

Individualized treatment effect (ITE) predictive modeling provides patient-specific estimates of treatment response to capture heterogeneity in how patients respond to interventions. Within the area of machine learning (ML), counterfactual models, often applied to observational data, have been developed within the potential outcome framework to estimate the outcomes of different treatment strategies by simulating “what-if” scenarios.[14–17] Deep counterfactual models build on this idea by incorporating complex clinical data and adjusting for measured confounding biases, such as patient demographics, comorbidities, prior treatments, and disease progression, to provide more accurate estimates of treatment effects.[18,19] By generating these counterfactual predictions, ML can aid clinicians to identify which therapies are most likely to benefit individual patients. However, most counterfactual models fail to account for unmeasured confounders, including clinical decision-making biases and hidden patient severity factors that influence treatment decisions. Additionally, the process of representation learning in deep counterfactual models can lead to the loss of important measured confounders,[20] which further impacts the ITE estimation.

We conducted a retrospective analysis of ICU patients at risk of progressing to IMV identified using the Vent.io respiratory failure risk prediction model.[1] To estimate the ITE of HFNC versus NIV as the initial respiratory support, we developed a deep counterfactual model RepFlow-CFR, a flow-based confounder adjustment framework that integrates representation learning, normalizing flows and counterfactual inference, which was designed to account for both measured and unmeasured confounders while improving causal effect estimation. We tested the hypothesis that the deep counterfactual model 1) would accurately identify the patients who would benefit from either HFNC or NIV, and 2) concordance between the model’s recommendations and the actual treatment administered is associated with improved clinical outcomes.

Results

In this section, we first identified high-risk patients using the Vent.io model and defined the early HFNC/NIV cohorts at two academic ICUs. We then evaluated the predictive performance of our RepFlow-CFR model in forecasting the need for IMV, both before and after site-specific fine-tuning. Next, we analyzed ITE to categorize patients by likely benefit from HFNC or NIV, and assessed differences in clinical characteristics and outcomes across groups. We further examined the top features contributing to ITE predictions using SHAP analysis. Finally, we evaluated whether concordance between model-recommended and actual treatments was associated with improved clinical outcomes, using both descriptive statistics and multivariable regression models across two sites.

Identification of Early HFNC/NIV Cohort

UCSD ICU cohort consists of 31,180 encounters, and UCI ICU cohort includes 3,290 encounters. Using the pretrained Vent.io, which was developed based on UCSD ICU cohort (Supplementary 5), we identified 5,685 encounters for UCSD high-risk cohort and 578 encounters for UCI high-risk cohort. The Vent.io achieved an AUC of 0.895, sensitivity of 0.603, specificity of 0.845 and positive predictive value of 0.177 at UCSD and an AUC of 0.842, sensitivity of 0.547, specificity of 0.836 and positive predictive value of 0.167 at UCI. Patient characteristics for these two ICU cohorts are reported in Table S5. We then filtered and selected patients who meet the criteria of first receiving either HFNC or NIV after Vent.io T0, with 1,956 encounters in UCSD early HFNC/NIV cohort and 169 encounters in UCI early HFNC/NIV cohort. Table 1 shows baseline characteristics in UCSD and UCI early HFNC/NIV cohorts. NIV-treated group has a higher Charlson Comorbidity Index (CCI) score and a higher prevalence of Chronic Obstructive Pulmonary Disease (COPD) and Congestive Heart Failure (CHF) compared to the HFNC-treated group at both sites.

IMV Predictive Performance

RepFlow-CFR achieved an AUC of 0.820 and a PR-AUC of 0.566 on the UCSD early HFNC/NIV cohort, which is comparable to the baseline CFR model (AUC: 0.821, PR-AUC: 0.571). However, when externally validated on the UCI early HFNC/NIV cohort, performance declined, with RepFlow-CFR achieving an AUC of 0.630 and a PR-AUC of 0.444, compared to CFR’s AUC of 0.656 and PR-AUC of 0.415. To improve generalizability, we fine-tuned both CFR and RepFlow-CFR using 25% of the UCI early HFNC/NIV cohort. After fine-tuning, RepFlow-CFR achieved an AUC of 0.727 and a PR-AUC of 0.553, while CFR achieved an AUC of 0.758 and a PR-AUC of 0.590 on the UCI site.

Predicted Individualized Treatment Effects

To categorize patients based on their predicted ITE, we defined absolute ITE ≤ 0.001 as indicating patients who are non-responders or indifferent to both treatments despite minor fluctuations in ITE estimates, and divided the patients into three groups: NIV preferred (ITE < −0.001), HFNC preferred (ITE > 0.001), or Indifferent (ITE between −0.001 and 0.001). Table 2 summarizes the characteristics of patients by the predicted ITE of RepFlow-CFR model. At UCSD, the HFNC preferred group exhibited the highest CCI and showed a greater prevalence of CHF and COPD compared to the NIV preferred group. Baseline physiological measures, including heart rate, respiratory rate, temperature, mean arterial pressure, and PaCO2, did not differ significantly across groups at either site. Rates of IMV were highest in the NIV preferred group at both UCSD (25.2%) and UCI (37.9%). Meanwhile, mortality and hospice outcomes were not significantly different across treatment groups at either site. The directionality of average treatment effects was maintained across both cohorts. While traditional group analysis may overlook important individual differences, the ITE identifies these variations to have more personalized treatment recommendations.

SHAP-Based Feature Interpretation of ITE Predictions

To better understand the ITE predictions made by the RepFlow-CFR model, we applied SHAP values to quantify the contribution of each input feature. Figure 1 presents the top 10 features ranked by mean absolute SHAP value for the UCSD and UCI cohorts. At the UCSD site (Figure 1a), the most influential features included severity scores (e.g., coaSOFA, HRSIRS), pre-hospital factors (e.g., preLOS), and clinical interventions such as anesthesia and pain medication use, suggesting that patient acuity and treatment context strongly influenced individualized treatment recommendations. At the UCI site (Figure 1b), laboratory values and organ-specific dysfunction, particularly Red_cell_width_delta, renalSOFA, and Sodium, were more dominant, indicating site-specific variation in model interpretation. While some features such as coaSOFA and Phosphate_delta appeared in both cohorts, the overall differences underscore the importance of local data characteristics in shaping treatment effect predictions.

Outcomes with treatment concordance

Our primary clinical outcome was the need for IMV, and our main secondary outcome was combination of in-hospital mortality and disposition to hospice. We first computed the outcome proportions based on whether patients received treatments in concordance with the ITE-predicted preferred treatment NIV or HFNC according to each method. By comparing concordance versus discordance groups across different ITE estimation approaches, we can assess whether aligning treatment decisions with model recommendations leads to improved outcomes.

As illustrated in Figures 2 and 3, both IMV and Mortality & Hospice outcomes were generally more favorable for patients who received treatments concordant with their predicted benefit across both sites. For IMV, RepFlow-CFR consistently showed lower rates in the concordant groups, including 19.27% vs. 27.06% for NIV at UCSD, and 13.33% vs. 20.00% for HFNC at UCI. Most methods followed similar trends, although the X-learner at UCI showed slightly higher IMV in the HFNC-concordant group. Mortality & Hospice outcomes also reflected this concordance benefit: RepFlow-CFR showed 27.22% vs. 33.72% for NIV at UCSD, and 20.00% vs. 40.00% for HFNC at UCI. Overall, concordant treatment was associated with better outcomes, with RepFlow-CFR demonstrating the most consistent improvements across settings and outcomes.

To evaluate whether concordance with the predicted treatment is independently associated with improved clinical outcomes, we developed two multivariable logistic regression models: one for predicting the need for IMV, and another for predicting a composite outcome of mortality or hospice. Both models adjusted for key clinical confounders, including age, gender, SOFA score, CCI score, and Vent.io score. As shown in Table 3, concordance with the treatment recommended by RepFlow-CFR was significantly associated with reduced risk of IMV at the UCSD site for both NIV and HFNC. Similarly, in the composite outcome model, RepFlow-CFR showed the strongest effect at the UCI site, with both NIV concordance and HFNC concordance significantly associated with lower mortality or hospice. These associations were not consistently observed with baseline methods such as Causal Forest, X-learner, or CFR. While some alternative models showed significant results at one site or for one treatment, RepFlow-CFR was the only method demonstrating robust and consistent benefit across both outcomes and treatment modalities.

Discussion

In this study, we developed and validated a deep counterfactual model, RepFlow-CFR, to estimate individualized treatment effects (ITE) of HFNC versus NIV as initial respiratory support for critically ill ICU patients at risk of requiring IMV. Our findings demonstrate that personalized treatment recommendations based on model-predicted ITEs were associated with improved clinical outcomes, including reduced rates of IMV and a combined endpoint of mortality or hospice.

These results underscore the limitations of relying solely on average treatment effects from randomized controlled trials (RCTs), which may obscure important heterogeneity in individual responses to therapy. While clinical guidelines provide important direction for specific patient population, such as those with acute exacerbation of COPD or cardiogenic pulmonary edema, many ICU patients present with complex and overlapping clinical features not addressed by these guidelines. As such, treatment decisions often rely on physician judgment, which may be subject to variability and bias.

By leveraging deep learning techniques, representation learning, and conditional normalizing flows, RepFlow-CFR offers a novel approach to estimate counterfactual outcomes while adjusting for both measured and unmeasured confounders. Treatment concordance with model-predicted recommendations was significantly associated with improved outcomes at both development and validation sites, suggesting that individualized treatment guidance could play a meaningful role in clinical decision-making. Compared to traditional ITE estimation methods such as Causal Forest, X-Learner, and baseline CFR, our model demonstrated more robust and consistent performance, particularly in the external cohort after fine-tuning.

The model’s performance in the validation cohort also demonstrates the feasibility of adapting counterfactual models to diverse patient populations with limited retraining. This is critical for real-world implementation, where patient characteristics, comorbidity profiles, and practice patterns may differ substantially across institutions. Moreover, the ability of RepFlow-CFR to integrate fine-grained clinical variables and learn nuanced patient representations positions it as a powerful tool for personalized critical care.

Importantly, our findings suggest that personalized, data-driven decision support could be complementary to existing guideline-based care. In cases where guidelines are ambiguous or do not fully account for patient complexity, counterfactual models may offer more precise treatment selection and help avoid unnecessary escalation or delay in care. Future research could focus on hybrid approaches that combine model-derived ITEs with clinical expertise and guideline constraints to develop interpretable, actionable tools for bedside decision-making.[24]

However, several limitations must be acknowledged. As a retrospective analysis, our findings are subject to biases inherent in observational data, including residual confounding due to unmeasured variables such as clinician intent or undocumented clinical indicators. Although we attempted to account for such biases using advanced modeling techniques, prospective validation is needed to establish causal conclusions. Additionally, both study sites were academic centers in the same geographic region, and further validation in community and international settings would enhance generalizability.

Methods

Study Design and Setting

We conducted a retrospective study using de-identified Electronic Health Record (EHR) data of all adult patients (≥18 years) who were admitted to the ICU at University of California San Diego Health (UCSD) between January 1, 2016, and December 31, 2023, and the University of California Irvine Health (UCI) between January 1, 2021, and August 15, 2022, as well as January 1, 2023, and August 31, 2024. Ethics approval was obtained from the University of California San Deigo Institutional Review Board.

Patients were included in the respiratory failure prediction analysis if they had an ICU stay of at least five hours, were not mechanically ventilated before ICU admission, and had recorded measurements of heart rate, blood pressure, and laboratory values prior to the prediction start time. If there were multiple admissions for a single patient, each ICU stay was treated as a separate encounter. Patients with a “Do Not Resuscitate” (DNR) order were excluded, and timestamps within 24 hours before and after surgery were omitted to avoid capturing surgery-related ventilation events. Monitoring for respiratory failure continued throughout each ICU stay until either mechanical ventilation was initiated or the patient was transferred out of the ICU. To ensure sufficient data collection, predictions began four hours after ICU admission and were updated hourly using the latest clinical data.

We used the pretrained Vent.io respiratory failure risk prediction model to identify high-risk ICU patients by thresholding the predicted state value. The model was trained using a custom 5-point labeling scale for mechanical ventilation (Table S1) to account for the various physiological states of respiratory failure, where a score ≥ 3 was defined as positive class and < 3 was control class. The high-risk threshold was selected at 60% sensitivity at the encounter level of the training data, based on clinical feedback, to minimize false positives. This threshold was then applied during model inference: if the predicted score met or exceeded the threshold, the patient was classified as high-risk for requiring IMV within the next 24 hours. A score below the threshold indicated that IMV was not predicted within the next 24 hours.

To evaluate the effect of initial treatment of HFNC and NIV on high-risk ICU patients, we defined Vent.io T0 as the time at which the patient’s risk score first crossed the high-risk threshold (score ≥ 3). We focused on the clinical data at Vent.io T0 and the initial respiratory support treatment administered after this timepoint, specifically including patients who first received either HFNC or NIV (referred to as early HFNC/NIV cohort). Treatment categories were further refined to distinguish specific HFNC and NIV interventions (Table S2). Figure 4 illustrates the cohort derivation process for the analysis of early HFNC and NIV as initial respiratory support following Vent.io-predicted high-risk timepoint (Vent.io T0). Using counterfactual modeling, we aimed to compare the effects of HFNC and NIV on the subsequent need for IMV and mortality.

Data Abstraction, Missingness, and Processing

We extracted data from EHR including 50 vital signs and laboratory measurements, 6 demographic features, 12 Systemic Inflammatory Response Syndrome (SIRS) and Sequential Organ Failure Assessment (SOFA) criteria, 12 medication categories, and 62 comorbidities (Table S3). The vital signs and laboratory measurements were grouped into one-hour time series bins to account for varying data sampling frequencies. Variables sampled more than once per hour were resampled into hourly bins using their median. Updates were made hourly with new data and if no new data were present, existing values were carried forward for up to 24 hours. All remaining missing values were replaced using mean imputation. We reported missing data on an hourly interval following the resampling process for the 50 vital signs and laboratory measurements (Table S4). In addition to the above clinical variables, we calculated 150 features derived from the 50 vital signs and laboratory measurements. For each vital sign and laboratory measurement, we derived baseline values (mean value measured over the previous 72 hours), local trends (change since last measurement), and the time since the variable was last measured (TSLM).

Model Development, Training and Evaluation

We proposed the RepFlow-CFR model, a flow-based confounder adjustment model that integrates representation learning, normalizing flows and counterfactual inference. Figure 5 presents the architecture overview of the RepFlow-CFR model, which contains three modules as follows.

Stage 0 (CFR-based representation learning): We utilized the counterfactual regression architecture (CFR)[18] that includes shared representation layers and two distinct heads for predicting outcomes under different treatments. The shared representation layers, based on the Vent.io architecture, include a TSLM layer for adjusting the importance of labs and vitals, followed by a feedforward neural network. By training the CFR model, the shared representation is encouraged to balance the distribution of measured confounders across treatment groups. The loss function is formulated as

$$[eqn]$$

where IPMG is the empirical probability metric (e.g. Wasserstein distance). LCFR is the prediction loss. λ denotes the trade-off parameter that balances prediction accuracy and representation distribution matching.

After training Stage 0, we assume that the learned representation ϕ captures sufficient information from measured covariates such that potential outcomes are conditionally independent of treatment assignment, given ϕ and a latent variable u representing unmeasured confounding as

$$[eqn]$$

where Y and A denote the outcome and treatment, respectively.

The observed distribution is defined as

$$[eqn]$$

The interventional distribution, which removes the influence of a on u, is

$$[eqn]$$

If p(u|ϕ, a) = p(u|ϕ), we will have p(Y(a) = y|ϕ) = p(y|ϕ, a). However, in observational studies, this assumption rarely holds due to treatment assignment bias or unmeasured confounding. Directly using p(y|ϕ, a) to estimate counterfactual outcomes would lead to biased inference. To address this, RepFlow-CFR includes two additional stages to explicitly model and account for this hidden bias.

Stage 1 (Modeling outcome distribution): We used a conditional normalizing flow (CNF)[21] to model the observed outcome distribution p(y|ϕ, a), where a ∈ {NIV, HFNC}. The CNF learns an invertible transformation $[eqn]$ that maps a standard normal latent variable U~N(0, I) to the outcome space. The loss function is formulated as

$$[eqn]$$

Stage 2 (Adjusting for hidden confounding): Since the learned representation ϕ may not satisfy unconfoundedness due to unmeasured confounding, we introduced a second CNF $[eqn]$ . It transforms a new latent variable $[eqn]$ to an interventional latent variable that approximates p(u|ϕ). The resulting latent sample is then passed through the Stage 1 transformation $[eqn]$ , which shifts the latent distribution p(u|ϕ, a) toward the interventional distribution p(u|ϕ). This adjustment allows us to account for hidden biases arising from factors like clinician decision-making, treatment selection bias, and unobserved patient severity. The loss function is formulated as

$$[eqn]$$

Inference stage: During inference, we sampled $[eqn]$ using the CNF from Stage 2 and mapped it to y using the CNF from stage 1, then averaged multiple predicted outcomes. ITE was obtained by the difference between the averaged outcomes under NIV and under HFNC.

We split the UCSD early HFNC/NIV cohort into an 80% training and 20% internal validation set. We applied Bayesian optimization[22] to tune hyperparameters including learning rates, weight regularization factor, number of hidden layers and flow depth. The optimization process was guided by validation performance across the CFR model and both CNF modules. Each module was trained separately using the Adam optimizer[23], with early stopping based on validation loss to mitigate overfitting. Model performance, including both predictive accuracy and ITE estimation quality, was evaluated on the entire UCSD early HFNC/NIV cohort. To assess generalizability, we further conducted external validation on an independent early HFNC/NIV cohort from UCI.

The predicted Individual Treatment Effect (ITE) was defined as the difference between the predicted probability of IMV under NIV and under HFNC, when each was given as the first intervention following the VentNet T0 timepoint. To assess predictive performance, we reported two standard metrics for IMV prediction: the Area Under the Receiver Operating Characteristic Curve (AUC) and the Area Under the Precision-Recall Curve (PR-AUC). ITE estimation quality was further evaluated by examining patient outcomes under treatment concordance. As baseline comparisons, we included commonly used data-driven ITE estimation methods in clinical settings, including Causal Forest[15], X-Learner[16], and the CFR model.

Supplementary Files

This is a list of supplementary files associated with this preprint. Click to download.

Supplementarymaterial.docx