Efficient Transfer Learning for Video-language Foundation Models

TL;DR

This paper introduces a parameter-efficient multi-modal spatio-temporal adapter for video-language models that improves transfer learning performance across various tasks while using minimal additional parameters.

Contribution

It proposes a novel, lightweight adapter and a spatio-temporal consistency constraint to enhance video-language model transferability and generalization with minimal parameter increase.

Findings

Achieves state-of-the-art results in zero-shot, few-shot, and fully-supervised tasks.

Uses only 2-7% of the original model's trainable parameters.

Enhances model generalization and reduces overfitting.

Abstract

Pre-trained vision-language models provide a robust foundation for efficient transfer learning across various downstream tasks. In the field of video action recognition, mainstream approaches often introduce additional modules to capture temporal information. Although the additional modules increase the capacity of model, enabling it to better capture video-specific inductive biases, existing methods typically introduce a substantial number of new parameters and are prone to catastrophic forgetting of previously acquired generalizable knowledge. In this paper, we propose a parameter-efficient Multi-modal Spatio-Temporal Adapter (MSTA) to enhance the alignment between textual and visual representations, achieving a balance between generalizable knowledge and task-specific adaptation. Furthermore, to mitigate over-fitting and enhance generalizability, we introduce a spatio-temporal description-guided consistency constraint.This constraint involves providing template inputs (e.g., "a video of \{\textbf{cls}\}") to the trainable language branch and LLM-generated spatio-temporal descriptions to the pre-trained language branch, enforcing output consistency between the branches. This approach reduces overfitting to downstream tasks and enhances the distinguishability of the trainable branch within the spatio-temporal semantic space. We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised learning. Compared to many state-of-the-art methods, our MSTA achieves outstanding performance across all evaluations, while using only 2-7\% of the trainable parameters in the original model.