Parameter-Efficient Transfer Learning for Audio-Visual-Language Tasks

TL;DR

This paper introduces LSTTA, a novel adapter-based method for efficient trimodal video understanding that models dense interactions across audio, visual, and language modalities, improving performance while reducing fine-tuning complexity.

Contribution

The paper proposes a new Long Short-Term Trimodal Adapter (LSTTA) architecture for trimodal tasks, enabling flexible, parameter-efficient learning across three modalities with novel modules for temporal and local interactions.

Findings

LSTTA outperforms existing trimodal methods on four benchmark tasks.

LSTTA effectively models dense interactions across audio, visual, and language modalities.

LSTTA is compatible with various pre-trained unimodal or bimodal models.

Abstract

The pretrain-then-finetune paradigm has been widely used in various unimodal and multimodal tasks. However, finetuning all the parameters of a pre-trained model becomes prohibitive as the model size grows exponentially. To address this issue, the adapter mechanism that freezes the pre-trained model and only finetunes a few extra parameters is introduced and delivers promising results. Most studies on adapter architectures are dedicated to unimodal or bimodal tasks, while the adapter architectures for trimodal tasks have not been investigated yet. This paper introduces a novel Long Short-Term Trimodal Adapter (LSTTA) approach for video understanding tasks involving audio, visual, and language modalities. Based on the pre-trained from the three modalities, the designed adapter module is inserted between the sequential blocks to model the dense interactions across the three modalities. Specifically, LSTTA consists of two types of complementary adapter modules, namely the long-term semantic filtering module and the short-term semantic interaction module. The long-term semantic filtering aims to characterize the temporal importance of the video frames and the short-term semantic interaction module models local interactions within short periods. Compared to previous state-of-the-art trimodal learning methods pre-trained on a large-scale trimodal corpus, LSTTA is more flexible and can inherit any powerful unimodal or bimodal models. Experimental results on four typical trimodal learning tasks show the effectiveness of LSTTA over existing state-of-the-art methods.