Task Matrices: Linear Maps for Cross-Model Finetuning Transfer
Darrin O' Brien, Dhikshith Gajulapalli, Eric Xia
TL;DR
This paper introduces task matrices, linear transformations that enable effective transfer from base to finetuned models across vision and language tasks, demonstrating their efficiency and generalizability.
Contribution
The paper proposes task matrices as a new method to capture linear encodings in models, outperforming linear probes and approaching finetuned performance.
Findings
Task matrices outperform linear probes in multiple datasets.
Linear encodings are validated across vision and language models.
Data-based approximation of encodings is efficient and generalizable.
Abstract
Results in interpretability suggest that large vision and language models learn implicit linear encodings when models are biased by in-context prompting. However, the existence of similar linear representations in more general adaptation regimes has not yet been demonstrated. In this work, we develop the concept of a task matrix, a linear transformation from a base to finetuned embedding state. We demonstrate that for vision and text models and ten different datasets, a base model augmented with a task matrix achieves results surpassing linear probes, sometimes approaching finetuned levels. Our results validate the existence of cross-layer linear encodings between pretrained and finetuned architectures. Moreover, we show that a data-based approximation for such encodings is both efficient and generalizable to multiple domains. We make our implementation publicly available.
Peer Reviews
Decision·Submitted to ICLR 2026
1. The experiments, conducted across multiple vision (CLIP ViT-B/32) and text (RoBERTa-large) models on ten different datasets, are comprehensive. The results consistently show that the task matrix approach surpasses the performance of linear probing, a standard baseline. 2. A key strength of the proposed method is its performance in data-scarce environments. The experiments show that task matrices maintain a significant performance advantage over linear probes when trained on only 20% of the da
1. **Insufficient Positioning Against Prior Work:** The core idea of using linear maps between embedding spaces has deep roots in prior work on concept learning (e.g., Paccanaro & Hinton, 2001; Mikolov et al., 2013). While the paper applies this to the new context of finetuning transfer, its novelty relative to this established literature is not clearly articulated. Furthermore, the connection to related methods like task arithmetic [Ilharco et al., 2022], which also manipulates model states, i
**Conceptual novelty:** The central idea -- that fine-tuning can be approximated as a linear transformation in representation space -- is novel and elegant. It reframes fine-tuning as a geometric relation rather than an optimization process, opening potential directions for efficient adaptation and cross-model understanding. **Empirical coverage:** The authors conduct an extensive empirical study spanning both vision and language domains, multiple model families, and diverse datasets. **Ab
**Experimental discussion and clarity.** While the empirical coverage is broad, the presentation and discussion of results are weak. Tables are numerous and scattered across the main and supplementary sections, but the commentary does not synthesize the findings into a coherent take-away. For instance, the reader is never clearly told \textit{whether} and \textit{under what conditions} the task matrices work best. A concise summary of results (e.g., performance trends across depth or domains) is
Method simplicity: the proposed task matrix construction method is straightforward and easy to implement. It consists essentially of a least-squares regression that learns a linear map between base and finetuned hidden representations. Generalization across multiple datasets and models: the approach is shown to generalize well to both vision and text models, and even across multiple datasets simultaneously. The authors demonstrate with extensive experiments that a single task matrix can support
No clear conceptual advantage over existing baselines: while the approach is elegant and achieves competitive performance, its advantages over standard techniques such as linear probing or low-rank adaptation remain unclear in the full-data setting. The authors describe task matrices as “lightweight” and “low-cost,” but do not provide runtime, FLOPs, or parameter count comparisons against LoRA, adapters, or full finetuning baselines. Potential equivalence to linear probing under specific condit
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Explainable Artificial Intelligence (XAI) · Topic Modeling