PVeRA: Probabilistic Vector-Based Random Matrix Adaptation

TL;DR

PVeRA introduces a probabilistic adaptation method for large models that improves parameter efficiency and performance by modifying low-rank matrices with probabilistic techniques, enabling better handling of input ambiguities.

Contribution

It proposes PVeRA, a novel probabilistic extension of VeRA, enhancing adaptation flexibility and performance on benchmark tasks.

Findings

PVeRA outperforms VeRA and other adapters on VTAB-1k.

Probabilistic modification improves handling of input ambiguities.

Code and benchmarking tools are publicly available.

Abstract

Large foundation models have emerged in the last years and are pushing performance boundaries for a variety of tasks. Training or even finetuning such models demands vast datasets and computational resources, which are often scarce and costly. Adaptation methods provide a computationally efficient solution to address these limitations by allowing such models to be finetuned on small amounts of data and computing power. This is achieved by appending new trainable modules to frozen backbones with only a fraction of the trainable parameters and fitting only these modules on novel tasks. Recently, the VeRA adapter was shown to excel in parameter-efficient adaptations by utilizing a pair of frozen random low-rank matrices shared across all layers. In this paper, we propose PVeRA, a probabilistic version of the VeRA adapter, which modifies the low-rank matrices of VeRA in a probabilistic manner. This modification naturally allows handling inherent ambiguities in the input and allows for different sampling configurations during training and testing. A comprehensive evaluation was performed on the VTAB-1k benchmark and seven adapters, with PVeRA outperforming VeRA and other adapters. Our code for training models with PVeRA and benchmarking all adapters is available https://github.com/leofillioux/pvera.