UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

TL;DR

UniAdapter introduces a parameter-efficient method for cross-modal transfer learning that unifies unimodal and multimodal adapters, achieving superior performance with minimal additional parameters across various vision-language tasks.

Contribution

The paper presents UniAdapter, a unified adapter framework that reduces parameters via weight sharing, enabling effective cross-modal transfer learning without full fine-tuning.

Findings

Outperforms state-of-the-art on multiple benchmarks.

Requires only 1-2% of the pre-trained model's parameters.

Achieves superior results on MSRVTT retrieval task.

Abstract

Large-scale vision-language pre-trained models have shown promising transferability to various downstream tasks. As the size of these foundation models and the number of downstream tasks grow, the standard full fine-tuning paradigm becomes unsustainable due to heavy computational and storage costs. This paper proposes UniAdapter, which unifies unimodal and multimodal adapters for parameter-efficient cross-modal adaptation on pre-trained vision-language models. Specifically, adapters are distributed to different modalities and their interactions, with the total number of tunable parameters reduced by partial weight sharing. The unified and knowledge-sharing design enables powerful cross-modal representations that can benefit various downstream tasks, requiring only 1.0%-2.0% tunable parameters of the pre-trained model. Extensive experiments on 6 cross-modal downstream benchmarks (including video-text retrieval, image-text retrieval, VideoQA, and VQA) show that in most cases, UniAdapter not only outperforms the state-of-the-arts, but even beats the full fine-tuning strategy. Particularly, on the MSRVTT retrieval task, UniAdapter achieves 49.7% recall@1 with 2.2% model parameters, outperforming the latest competitors by 2.0%. The code and models are available at https://github.com/RERV/UniAdapter.