Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge

TL;DR

Layer-Aware Task Arithmetic (LATA) improves multi-task learning and task forgetting in large language models by assigning layer-specific weights to disentangle task-specific knowledge from instruction-following behavior, enhancing performance and model utility.

Contribution

The paper introduces LATA, a novel layer-wise weighting method that better isolates task-specific knowledge from instruction-following components in LLMs.

Findings

LATA outperforms existing methods in multi-task learning accuracy.

LATA achieves superior task forgetting with minimal output quality degradation.

Layer-wise analysis effectively disentangles task-specific and instruction-following knowledge.

Abstract

Large language models (LLMs) demonstrate strong task-specific capabilities through fine-tuning, but merging multiple fine-tuned models often leads to degraded performance due to overlapping instruction-following components. Task Arithmetic (TA), which combines task vectors derived from fine-tuning, enables multi-task learning and task forgetting but struggles to isolate task-specific knowledge from general instruction-following behavior. To address this, we propose Layer-Aware Task Arithmetic (LATA), a novel approach that assigns layer-specific weights to task vectors based on their alignment with instruction-following or task-specific components. By amplifying task-relevant layers and attenuating instruction-following layers, LATA improves task learning and forgetting performance while preserving overall model utility. Experiments on multiple benchmarks, including WikiText-2, GSM8K, and HumanEval, demonstrate that LATA outperforms existing methods in both multi-task learning and selective task forgetting, achieving higher task accuracy and alignment with minimal degradation in output quality. Our findings highlight the importance of layer-wise analysis in disentangling task-specific and general-purpose knowledge, offering a robust framework for efficient model merging and editing.