# Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance

## TL;DR

This paper introduces CPI-FT, a novel fine-tuning framework that isolates core parameters for each task, improving multi-task learning performance and reducing interference in large language models.

## Contribution

The paper proposes a new Core Parameter Isolation Fine-Tuning method that identifies and isolates core parameters per task, enhancing multi-task fine-tuning effectiveness.

## Key findings

- Significantly reduces task interference and forgetting.
- Outperforms vanilla multi-task and multi-stage fine-tuning baselines.
- Demonstrates effectiveness on multiple public benchmarks.

## Abstract

Supervised fine-tuning (SFT) is a pivotal approach to adapting large language models (LLMs) for downstream tasks; however, performance often suffers from the ``seesaw phenomenon'', where indiscriminate parameter updates yield progress on certain tasks at the expense of others. To address this challenge, we propose a novel \emph{Core Parameter Isolation Fine-Tuning} (CPI-FT) framework. Specifically, we first independently fine-tune the LLM on each task to identify its core parameter regions by quantifying parameter update magnitudes. Tasks with similar core regions are then grouped based on region overlap, forming clusters for joint modeling. We further introduce a parameter fusion technique: for each task, core parameters from its individually fine-tuned model are directly transplanted into a unified backbone, while non-core parameters from different tasks are smoothly integrated via Spherical Linear Interpolation (SLERP), mitigating destructive interference. A lightweight, pipelined SFT training phase using mixed-task data is subsequently employed, while freezing core regions from prior tasks to prevent catastrophic forgetting. Extensive experiments on multiple public benchmarks demonstrate that our approach significantly alleviates task interference and forgetting, consistently outperforming vanilla multi-task and multi-stage fine-tuning baselines.