W2T: LoRA Weights Already Know What They Can Do

TL;DR

This paper introduces W2T, a method that transforms LoRA weights into a canonical form to directly infer model behavior and performance without running the models or needing training data.

Contribution

W2T provides a novel canonicalization process for LoRA weights, enabling direct interpretation of model updates and performance prediction from weight representations.

Findings

W2T achieves high accuracy in attribute classification.

W2T effectively predicts model performance.

W2T enables reliable adapter retrieval.

Abstract

Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token.