AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition

TL;DR

AdaRing introduces a novel tensor ring decomposition method for ultra-light, parameter-efficient vision-language model adaptation, significantly reducing training parameters while maintaining state-of-the-art performance across various tasks.

Contribution

It proposes a cross-layer tensor ring decomposition framework that exploits redundancy and diversity among adapters for ultra-light VLM fine-tuning.

Findings

Achieves state-of-the-art performance on multiple vision-language tasks.

Reduces average training parameters by 90%.

Effectively models cross-layer redundancy and diversity among adapters.

Abstract

Adapter-based fine-tuning has gained remarkable attention in adapting large pre-trained vision language models (VLMs) for a wide range of downstream tasks efficiently. In this paradigm, only the inserted adapters are fine-tuned, without the need for training the original VLM backbone. Existing works scale adapters by integrating them into every layer of VLMs to increase the capacity of adapters. However, these methods face two primary limitations: 1) limited compression rate due to ignoring cross-layer redundancy, and 2) limited representational capacity across homogeneous adapters. In this paper, we propose a novel vision-language fine-tuning framework based on cross-layer tensor ring decomposition (TRD) with the integration and collaboration of diverse adapters, called AdaRing, achieving ultra-light parameter-efficient adaptation of VLMs on various tasks. To remove the high redundancy that exists among adapters across layers, we exploit the tensor-level low-rankness to formulate adapters as layer-shared tensor cores and layer-specific slices. Moreover, guided by generalization-aware fine-tuning, diverse rank-driven adapters cooperate to handle tasks that require different representations. Our experiments show that the proposed AdaRing achieves the state-of-the-art performance while reducing average training parameters by 90%.