Training-Free Adaptation of New-Generation LLMs using Legacy Clinical Models

TL;DR

The paper introduces CAPT, a training-free method for adapting new-generation language models to clinical tasks by ensembling with legacy models, improving clinical relevance without retraining.

Contribution

CAPT enables training-free adaptation of modern language models to clinical domains using existing models, outperforming previous ensembling methods across multiple tasks.

Findings

CAPT outperforms individual models and state-of-the-art ensembling methods by large margins.

CAPT enhances clinical specificity and reduces errors in language generation.

CAPT benefits healthcare settings with limited computational resources.

Abstract

Adapting language models to the clinical domain through continued pretraining and instruction tuning requires costly retraining for each new model generation. We propose Cross-Architecture Proxy Tuning (CAPT), a model-ensembling approach that enables training-free adaptation of state-of-the-art general-domain models using existing clinical models. CAPT supports models with disjoint vocabularies, leveraging contrastive decoding to selectively inject clinically relevant signals while preserving the general-domain model's reasoning and fluency. On six clinical classification and text-generation tasks, CAPT with a new-generation general-domain model and an older-generation clinical model consistently outperforms both models individually and state-of-the-art ensembling approaches (average +17.6\% over UniTE, +41.4\% over proxy tuning across tasks). Through token-level analysis and physician case studies, we demonstrate that CAPT amplifies clinically actionable language, reduces context errors, and increases clinical specificity. This technique especially benefits healthcare institutions with constrained computational capacity that cannot support iterative clinical training and want to adopt emerging general-domain model advances.