# Development and validation of a generalisable machine learning algorithm for identifying interstitial lung disease cohorts: a retrospective cohort study

## TL;DR

This study developed a machine learning algorithm to accurately identify interstitial lung disease cases using electronic health records, outperforming traditional rule-based methods.

## Contribution

The Universal ILD Classifier is a novel machine learning algorithm that improves identification of interstitial lung disease using EHR data.

## Key findings

- The Universal ILD Classifier achieved a PPV of 0.67 and ROC-AUC of 0.96, outperforming rule-based methods.
- The algorithm demonstrated robust generalisability across three independent validation sites.
- It showed high sensitivity (0.97) and F1-score (0.79), making it suitable for large-scale ILD research.

## Abstract

Large electronic databases are powerful tools for studying rare diseases, however accurate Interstitial Lung Disease (ILD) classification remains challenging. Rule-based approaches rely heavily on diagnostic codes—unreliable markers of ILD. We aimed to develop and externally validate an ILD classification algorithm that robustly identifies prevalent cases using routinely captured electronic health record (EHR) data.

In this retrospective model development and validation study, we used EHR data from the UC Health Data Warehouse, a multi-institutional dataset from six academic centres in California, USA (2012–2024). Data from individuals ≥18 years with ≥ five encounters were included. We developed the Universal ILD Classifier, a machine learning model developed on standardised EHR data elements from UC San Francisco (January 1, 1981–January 6, 2025). The algorithm was converted to an EHR-agnostic common data model, to enable external validation across three independent sites (UC Irvine, Los Angeles, and San Diego; January 1, 2012–April 30, 2025). Features included diagnostic and procedure codes, laboratory results, medications, demographics, and utilisation metrics. The main outcome was algorithm performance assessed by positive predictive value (PPV), sensitivity, F1-score, and receiver operative characteristic–area under the curve (ROC-AUC). Performance was compared with two widely used rule-based classification methods.

The Universal ILD Classifier, developed on data from 203,976 patients and validated on data at three independent sites (N = 250 per site), demonstrated robust generalisability, achieving average PPV = 0.67 (0.58–0.72), sensitivity = 0.97 (0.94–0.99), F1-score = 0.79 (0.72–0.84), and ROC-AUC = 0.96 (0.94–0.97). It consistently outperformed both rule-based methods, which had PPVs = 0.55 (0.50–0.59) and 0.67 (0.61–0.73), sensitivities = 0.98 (0.96–0.99) and 0.59 (0.53–0.64), F1-scores = 0.71 (0.66–0.74) and 0.63 (0.57–0.68), and ROC-AUCs = 0.80 (0.78–0.82) and 0.73 (0.70–0.76) respectively.

Accurate patient identification is essential for epidemiological studies and ILD clinical trials. The Universal ILD Classifier leverages commonly available EHR data and outperforms rule-based approaches, supporting more reliable large-scale ILD research and offering a foundation for further refinement with additional features. Limitations inherent to retrospective EHR analyses, including misclassification, residual confounding, and limited generalisability, may have influenced effect estimates and should be considered when interpreting these findings.

10.13039/100001003Boehringer Ingelheim Pharmaceuticals, Inc. (BIPI).