MedCL-Bench: Benchmarking stability-efficiency trade-offs and scaling in biomedical continual learning

TL;DR

MedCL-Bench introduces a comprehensive benchmarking framework for evaluating stability and efficiency trade-offs in biomedical continual learning, highlighting the impact of task order, model strategies, and task types on performance.

Contribution

It provides the first unified, task-diverse benchmark for biomedical NLP continual learning, including standardized protocols and analysis of various strategies across multiple datasets.

Findings

Sequential fine-tuning causes catastrophic forgetting.

Parameter-isolation offers the best retention per GPU-hour.

Forgetting varies by task type, with multi-label classification most vulnerable.

Abstract

Medical language models must be updated as evidence and terminology evolve, yet sequential updating can trigger catastrophic forgetting. Although biomedical NLP has many static benchmarks, no unified, task-diverse benchmark exists for evaluating continual learning under standardized protocols, robustness to task order and compute-aware reporting. We introduce MedCL-Bench, which streams ten biomedical NLP datasets spanning five task families and evaluates eleven continual learning strategies across eight task orders, reporting retention, transfer, and GPU-hour cost. Across backbones and task orders, direct sequential fine-tuning on incoming tasks induces catastrophic forgetting, causing update-induced performance regressions on prior tasks. Continual learning methods occupy distinct retention-compute frontiers: parameter-isolation provides the best retention per GPU-hour, replay offers strong protection at higher cost, and regularization yields limited benefit. Forgetting is task-dependent, with multi-label topic classification most vulnerable and constrained-output tasks more robust. MedCL-Bench provides a reproducible framework for auditing model updates before deployment.