AdaRank: Disagreement Based Module Rank Prediction for Low-rank Adaptation

TL;DR

AdaRank introduces a disagreement-based method to predict optimal module ranks for low-rank adaptation, improving transfer learning efficiency without altering pretraining or requiring extra objectives.

Contribution

It proposes a novel disagreement-based technique for module rank prediction that enhances adaptation performance while preserving the original training process.

Findings

AdaRank outperforms uniform rank methods on unseen data.

The method requires no additional objectives or regularizers.

It maintains the integrity of pretraining and adaptation stages.

Abstract

With the rise of language and multimodal models of ever-increasing size, pretraining a general-purpose foundational model and adapting it to downstream tasks has become common practice. To this end, adaptation efficiency can be a critical bottleneck given the large model sizes, hence efficient finetuning methods such as LoRA have become prevalent. However, LoRA is typically applied with the same rank across all model layers, despite mounting evidence from transfer learning literature that during finetuning, later layers diverge more from pretrained weights. Inspired by the theory and observations around feature learning and module criticality, we develop a simple model disagreement based technique to predict the rank of a given module relative to the other modules. Empirically, AdaRank generalizes notably better on unseen data than using uniform ranks with the same number of parameters. Compared to prior work, AdaRank has the unique advantage of leaving the pretraining and adaptation stages completely intact: no need for any additional objectives or regularizers, which can hinder adaptation accuracy and performance. Our code is publicly available at https://github.com/google-research/google-research/tree/master/adaptive_low_rank.