# Inferring tumor immune microenvironment -related risk states from pretreatment H&E pathomics and clinical biomarkers to predict checkpoint inhibitor pneumonitis in advanced NSCLC: a multicenter multimodal study

## TL;DR

This study uses AI-driven analysis of H&E slides and clinical data to predict the risk of checkpoint inhibitor pneumonitis in lung cancer patients before treatment.

## Contribution

A novel multimodal model combining pathomics and clinical biomarkers to predict pneumonitis risk in NSCLC patients.

## Key findings

- The multimodal fusion model achieved the highest AUC scores (up to 0.930) for predicting CIP risk.
- Pathomics models outperformed clinical models in predicting pneumonitis risk.
- Integration of H&E pathomics and clinical data enables accurate pretreatment risk stratification.

## Abstract

Checkpoint inhibitor pneumonitis (CIP) is a rare but potentially fatal immune-related adverse event (irAE) that can interrupt immune checkpoint blockade in non-small cell lung cancer (NSCLC). With no validated pretreatment biomarkers and a diagnosis largely made by exclusion, upfront risk stratification is required. Recent advances in artificial intelligence (AI)-driven pathomics have made it feasible to infer tumor immune microenvironment (TIME)-relevant risk states in patients with NSCLC. Accordingly, we leveraged hematoxylin and eosin(H&E)-based digital pathomics combined with clinical variables to interrogate the TIME in patients who developed CIP and to enable pretreatment and early prediction of CIP.

In this retrospective study, 346 eligible patients from three hospitals were screened consecutively between January 2022 and January 2025. Patients were divided into CIP and non-CIP groups according to whether CIP occurred at a prespecified observation endpoint. We first developed a pathomics model that employed convolutional neural networks (CNNs) combined with multi-instance learning (MIL) to generate predictions at both the patch and whole slide image (WSI) levels on H&E-stained slides. Separately, we constructed a clinical model using logistic regression (LR) to process the structured clinical data accompanying each case. Subsequently, pathological and clinical information were integrated, where modeling was advanced from modality-specific feature learning to cross-modal representation learning, and final predictive modeling was completed. The predictive performance of different models was evaluated using the area under the Receiver Operating Characteristic (ROC) curve and benchmarked against unimodal models and standard ensemble methods.

When the models were evaluated across both internal validation and external test datasets, the pathomics model demonstrated noticeably stronger performance than the clinical approach, achieving area under the curve (AUC) scores of 0.916, 0.875(test 1), and 0.843(test 2), respectively, while the clinical model posted more modest results of 0.880, 0.569(test 1), and 0.594(test 2). The most significant outcome, however, emerged from the multimodal fusion model, which produced the strongest results of all, with performance metrics of 0.930, 0.919(test 1), and 0.905(test 2) in the validation and test phases, respectively.

Pretreatment H&E-derived pathomics, integrated with baseline clinical biomarkers, enable accurate prediction of CIP risk in locally advanced or metastatic NSCLC. This framework supports proactive surveillance and individualized immune checkpoint inhibitor (ICI) strategies and provides a scalable route to decode TIME-relevant states from routine pathology.