Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models

TL;DR

This paper introduces Learning on LoRAs (LoL), a novel approach that processes low-rank weight decompositions of large models using symmetry-aware invariant and equivariant architectures to predict performance and other attributes.

Contribution

The paper develops symmetry-aware models for processing LoRA weights, addressing inherent parameter symmetries, and demonstrates their effectiveness in predicting model performance and attributes.

Findings

LoL models accurately predict CLIP scores and finetuning data attributes.

Symmetry-aware architectures outperform baseline methods.

Effective processing of low-rank weight decompositions for various tasks.

Abstract

Low-rank adaptations (LoRAs) have revolutionized the finetuning of large foundation models, enabling efficient adaptation even with limited computational resources. The resulting proliferation of LoRAs presents exciting opportunities for applying machine learning techniques that take these low-rank weights themselves as inputs. In this paper, we investigate the potential of Learning on LoRAs (LoL), a paradigm where LoRA weights serve as input to machine learning models. For instance, an LoL model that takes in LoRA weights as inputs could predict the performance of the finetuned model on downstream tasks, detect potentially harmful finetunes, or even generate novel model edits without traditional training methods. We first identify the inherent parameter symmetries of low rank decompositions of weights, which differ significantly from the parameter symmetries of standard neural networks. To efficiently process LoRA weights, we develop several symmetry-aware invariant or equivariant LoL models, using tools such as canonicalization, invariant featurization, and equivariant layers. We finetune thousands of text-to-image diffusion models and language models to collect datasets of LoRAs. In numerical experiments on these datasets, we show that our LoL architectures are capable of processing low rank weight decompositions to predict CLIP score, finetuning data attributes, finetuning data membership, and accuracy on downstream tasks.