Countering Catastrophic Forgetting of Large Language Models for Better Instruction Following via Weight-Space Model Merging

TL;DR

This paper introduces a weight-space model merging method to adapt large language models to the medical domain, effectively mitigating catastrophic forgetting and maintaining instruction-following capabilities.

Contribution

The study proposes a novel model merging framework that combines general and clinical LLMs, improving domain adaptation efficiency and performance in clinical tasks.

Findings

Merged models outperform baseline fine-tuning on medical benchmarks.

Model merging preserves instruction-following ability while adapting to clinical data.

Training efficiency is improved, matching fully fine-tuned models with less supervision.

Abstract

Large language models have been adopted in the medical domain for clinical documentation to reduce clinician burden. However, studies have reported that LLMs often "forget" a significant amount of instruction-following ability when fine-tuned using a task-specific medical dataset, a critical challenge in adopting general-purpose LLMs for clinical applications. This study presents a model merging framework to efficiently adapt general-purpose LLMs to the medical domain by countering this forgetting issue. By merging a clinical foundation model (GatorTronLlama) with a general instruct model (Llama-3.1-8B-Instruct) via interpolation-based merge methods, we seek to derive a domain-adapted model with strong performance on clinical tasks while retaining instruction-following ability. Comprehensive evaluation across medical benchmarks and five clinical generation tasks (e.g., radiology and discharge summarization) shows that merged models can effectively mitigate catastrophic forgetting, preserve clinical domain expertise, and retain instruction-following ability. In addition, our model merging strategies demonstrate training efficiency, achieving performance on par with fully fine-tuned baselines under severely constrained supervision (e.g., 64-shot vs. 256-shot). Consequently, weight-space merging constitutes a highly scalable solution for adapting open-source LLMs to clinical applications, facilitating broader deployment in resource-constrained healthcare environments.