Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature

TL;DR

This paper introduces a novel, data-free regularization method based on Kronecker-Factored Approximate Curvature to improve task vector disentanglement in task arithmetic, enhancing modularity and robustness without external data.

Contribution

It proposes a dataless regularization technique using curvature matrix approximation, achieving state-of-the-art results and constant complexity in task addition.

Findings

Achieves state-of-the-art results in task addition and negation.

Eliminates the need for held-out tuning.

Promotes robustness to task vector rescaling.

Abstract

Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors; however, existing approaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, eliminating the need for held-out tuning.