A Domain Incremental Continual Learning Benchmark for ICU Time Series Model Transportability

TL;DR

This paper introduces a benchmark for evaluating how well machine learning models trained in one hospital region can adapt to new regions in ICU time series data, addressing generalizability and transferability issues.

Contribution

It proposes a novel domain incremental learning benchmark for ICU time series, framing regional model transfer as a continual learning problem and evaluating methods like data replay and EWC.

Findings

The benchmark reveals significant regional differences in ICU data distributions.

Data replay and EWC show varying effectiveness in maintaining performance across regions.

Abstract

In recent years, machine learning has made significant progress in clinical outcome prediction, demonstrating increasingly accurate results. However, the substantial resources required for hospitals to train these models, such as data collection, labeling, and computational power, limit the feasibility for smaller hospitals to develop their own models. An alternative approach involves transferring a machine learning model trained by a large hospital to smaller hospitals, allowing them to fine-tune the model on their specific patient data. However, these models are often trained and validated on data from a single hospital, raising concerns about their generalizability to new data. Our research shows that there are notable differences in measurement distributions and frequencies across various regions in the United States. To address this, we propose a benchmark that tests a machine learning model's ability to transfer from a source domain to different regions across the country. This benchmark assesses a model's capacity to learn meaningful information about each new domain while retaining key features from the original domain. Using this benchmark, we frame the transfer of a machine learning model from one region to another as a domain incremental learning problem. While the task of patient outcome prediction remains the same, the input data distribution varies, necessitating a model that can effectively manage these shifts. We evaluate two popular domain incremental learning methods: data replay, which stores examples from previous data sources for fine-tuning on the current source, and Elastic Weight Consolidation (EWC), a model parameter regularization method that maintains features important for both data sources.