Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models
Guillermo Ortiz-Jimenez, Alessandro Favero, Pascal Frossard

TL;DR
This paper investigates task arithmetic in vision-language models, revealing that weight disentanglement and tangent space fine-tuning significantly enhance model editing, supported by theoretical and empirical NTK analyses.
Contribution
It demonstrates that weight disentanglement is key to task arithmetic effectiveness and shows that tangent space fine-tuning improves performance, linking these phenomena to NTK properties.
Findings
Weight disentanglement arises during pre-training.
Tangent space fine-tuning amplifies weight disentanglement.
NTK eigenfunctions relate to task arithmetic effectiveness.
Abstract
Task arithmetic has recently emerged as a cost-effective and scalable approach to edit pre-trained models directly in weight space: By adding the fine-tuned weights of different tasks, the model's performance can be improved on these tasks, while negating them leads to task forgetting. Yet, our understanding of the effectiveness of task arithmetic and its underlying principles remains limited. We present a comprehensive study of task arithmetic in vision-language models and show that weight disentanglement is the crucial factor that makes it effective. This property arises during pre-training and manifests when distinct directions in weight space govern separate, localized regions in function space associated with the tasks. Notably, we show that fine-tuning models in their tangent space by linearizing them amplifies weight disentanglement. This leads to substantial performance…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Advanced Neural Network Applications · Explainable Artificial Intelligence (XAI)
MethodsNeural Tangent Kernel
