Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms

TL;DR

This paper introduces Queryable LoRA, a parameter-efficient fine-tuning method that uses shared, content-dependent low-rank update atoms with instruction-regularization, improving adaptability and stability in large neural networks.

Contribution

It proposes a novel routing mechanism over shared low-rank update atoms with instruction-regularization, enabling dynamic, input-dependent model updates while maintaining efficiency.

Findings

Improved test performance over standard LoRA in experiments.

Enhanced training stability with the new routing approach.

Maintains parameter efficiency comparable to existing methods.

Abstract

We present a data-adaptive method for parameter-efficient fine-tuning of large neural networks. Standard low-rank adaptation methods improve efficiency by restricting each layer update to a fixed low-rank form, but this static parameterization can be too rigid when the appropriate correction depends on the input and on the evolving depth-wise computation of the network. Our approach replaces a purely layer-local adapter with a shared queryable memory of low-rank update atoms. For each block of layers, the model forms a query from the current low-rank state and a running summary of previous blocks, uses this query to retrieve a content-dependent combination of shared update components via attention, and applies the resulting routed operator within the low-rank bottleneck. In this way, the method retains the efficiency and scalability of low-rank adaptation while allowing the effective update to vary across inputs and to share reusable structure across layers. The resulting architecture provides a principled middle ground between static LoRA-style updates and fully generated parameter updates: it remains compact and parameter-efficient while supporting dynamic, context-sensitive adaptation. Further, we incorporate instruction-regularization by augmenting routing logits with a language-induced prior over update atoms, thereby biasing the selection of low-rank transformations toward semantically relevant directions without generating unconstrained parameter updates. Experiments on noisy non-linear regression tasks and LLM fine-tuning suggest that this queryable update-memory formulation can improve final test performance and training stability compared to standard low-rank adaptation, while using a comparable number of trainable parameters.