Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment

TL;DR

This paper introduces a few-shot orthogonal alignment method that enables effective transfer of task vectors between independently pre-trained models, improving transfer accuracy while preserving modularity.

Contribution

The paper proposes a novel orthogonal alignment technique for cross-model task vector transfer, addressing limitations of traditional task arithmetic in independently pre-trained models.

Findings

Improved transfer accuracy over direct task vector application.

Achieved performance comparable to few-shot fine-tuning.

Maintained modularity and reusability of task vectors.

Abstract

Task arithmetic enables efficient model editing by representing task-specific changes as vectors in parameter space. Task arithmetic typically assumes that the source and target models are initialized from the same pre-trained parameters. This assumption limits its applicability in cross-model transfer settings, where models are independently pre-trained on different datasets. To address this challenge, we propose a method based on few-shot orthogonal alignment, which aligns task vectors to the parameter space of a differently pre-trained target model. These transformations preserve key properties of task vectors, such as norm and rank, and are learned using only a small number of labeled examples. We evaluate the method using two Vision Transformers pre-trained on YFCC100M and LAION400M, and test on eight classification datasets. Experimental results show that our method improves transfer accuracy over direct task vector application and achieves performance comparable to few-shot fine-tuning, while maintaining the modularity and reusability of task vectors. Our code is available at https://github.com/kawakera-lab/CrossModelTransfer.