Transporting Task Vectors across Different Architectures without Training

TL;DR

This paper introduces Theseus, a training-free method for transferring task-specific updates across models of different widths by aligning their intermediate representations, enabling effective adaptation without retraining.

Contribution

The work presents a novel functional matching approach for cross-architecture task transfer, extending beyond identical models to heterogeneous widths without additional training.

Findings

Theseus improves performance across vision and language models of different widths.

The method achieves stable transfer by aligning representations via orthogonal Procrustes analysis.

It outperforms baselines without requiring extra training or backpropagation.

Abstract

Adapting large pre-trained models to downstream tasks often produces task-specific parameter updates that are expensive to relearn for every model variant. While recent work has shown that such updates can be transferred between models with identical architectures, transferring them across models of different widths remains unexplored. In this work, we introduce Theseus, a training-free method for transporting task updates across heterogeneous-width models. Rather than matching parameters, we characterize a task update by the functional effect it induces on intermediate representations. We formalize task-vector transport as a functional matching problem on observed activations and show that, after aligning representation spaces via orthogonal Procrustes analysis, it admits a stable closed-form solution that preserves the geometry of the update. We evaluate Theseus on vision and language models across different widths, showing consistent improvements over baselines without additional training or backpropagation. Our results show that task updates can be meaningfully transferred across architectures when task identity is defined functionally rather than parametrically. Code is available at https://github.com/apanariello4/merge-and-rebase.