Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning

TL;DR

This paper introduces a partition-based multi-stage fine-tuning framework for large language models that leverages inter-domain synergies and minimizes negative transfer, improving multi-domain adaptation.

Contribution

The paper presents a novel partitioning strategy and theoretical analysis for multi-domain LLM fine-tuning, addressing inter-domain interference effectively.

Findings

Outperforms state-of-the-art baselines in language understanding tasks

Provides theoretical generalization bounds for the proposed framework

Effectively balances domain discrepancy and synergy during fine-tuning

Abstract

Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due to inter-domain interference. To overcome this challenge, we propose a partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strategically partitions domains into subsets (stages) by balancing domain discrepancy, synergy, and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify our partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines.