Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformers

TL;DR

This paper introduces CLoRA, a novel low-rank adaptation method for vision transformers that balances parameter efficiency and learning performance through shared spaces and diversity regularization.

Contribution

The paper proposes CLoRA, which combines base-space sharing and sample-agnostic diversity enhancement to improve fine-tuning of vision transformers efficiently.

Findings

Outperforms state-of-the-art methods in image and point cloud tasks.

Requires fewer GFLOPs for point cloud analysis.

Achieves a better balance between performance and parameter efficiency.

Abstract

Low-rank adaptation (LoRA) has achieved remarkable success in fine-tuning pre-trained vision transformers for various downstream tasks. Existing studies mainly focus on exploring more parameter-efficient strategies or more effective representation learning schemes. However, these methods either sacrifice fine-tuning performance or introduce excessive trainable parameters, failing to strike a balance between learning performance and parameter efficiency. To address this problem, we propose a novel tuning method named collaborative low-rank adaptation (CLoRA) in this paper. CLoRA consists of base-space sharing and sample-agnostic diversity enhancement (SADE) components. To maintain parameter efficiency while expanding the learning capacity of low-rank modules (LRMs), base-space sharing allows all LRMs to share a set of down/up-projection spaces. In CLoRA, the low-rank matrices obtained from the shared spaces collaboratively construct each LRM. Since the representations extracted by these matrices may contain redundant information, SADE is employed to regularize the similarities among them to encourage diverse representations in the training process. We conduct extensive experiments on widely used image and point cloud datasets to evaluate the performance of CLoRA. Experimental results demonstrate that CLoRA strikes a better balance between learning performance and parameter efficiency, while requiring the fewest GFLOPs for point cloud analysis, compared with the state-of-the-art methods.