Null-LoRA: Low-Rank Adaptation on Null Space

TL;DR

Null-LoRA introduces a null space-based low-rank adaptation method that improves parameter efficiency and effectiveness in fine-tuning large models by leveraging the null space of pre-trained models.

Contribution

It proposes Null-LoRA, a novel fine-tuning approach that constrains updates within the null space to reduce redundancy and enhance adaptation efficiency.

Findings

Null-LoRA outperforms existing methods with fewer parameters.

It achieves superior results in image-text retrieval tasks.

It improves visual question answering performance.

Abstract

Parameter-efficient fine-tuning methods have gained considerable popularity for adapting large-scale models to downstream tasks, particularly LoRA and its variants. Existing methods perform low-rank adaptation over the full parameter space. However, fine-tuning within a subspace can achieve comparable effectiveness. Inspired by the observation that pre-trained models possess non-trivial null spaces, we propose Null-space based Low-Rank Adaptation (Null-LoRA). Null-LoRA effectively reduces redundancy and enhances effective rank by freezing portions of the low-rank matrices. To further improve parameter efficiency, Null-LoRA constrains the entire incremental update within the null space, maximizing the utilization of incremental updates to adapt to new task paradigms. Null-LoRA surpasses the state of the art with fewer parameters in extensive experiments across image-text retrieval and visual question answering tasks.