Less is More: Undertraining Experts Improves Model Upcycling

TL;DR

This paper investigates how prolonged expert fine-tuning can harm model upcycling performance, revealing that early stopping strategies can enhance the reuse of models in multi-task systems.

Contribution

It challenges the assumption that longer fine-tuning always improves transfer, showing that early stopping can prevent memorization issues and improve upcycling results.

Findings

Long fine-tuning degrades merging performance.

Memorization of difficult examples causes degradation.

Early stopping improves upcycling performance.

Abstract

Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets. This has led to a proliferation of expert models and adapters, often shared via platforms like HuggingFace and AdapterHub. To leverage these resources, numerous model upcycling methods have emerged, enabling the reuse of fine-tuned models in multi-task systems. A natural pipeline has thus formed to harness the benefits of transfer learning and amortize sunk training costs: models are pre-trained on general data, fine-tuned on specific tasks, and then upcycled into more general-purpose systems. A prevailing assumption is that improvements at one stage of this pipeline propagate downstream, leading to gains at subsequent steps. In this work, we challenge that assumption by examining how expert fine-tuning affects model upcycling. We show that long fine-tuning of experts that optimizes for their individual performance leads to degraded merging performance, both for fully fine-tuned and LoRA-adapted models, and to worse downstream results when LoRA adapters are upcycled into MoE layers. We trace this degradation to the memorization of a small set of difficult examples that dominate late fine-tuning steps and are subsequently forgotten during merging. Finally, we demonstrate that a task-dependent aggressive early stopping strategy can significantly improve upcycling performance.