Parameter Efficient Multimodal Transformers for Video Representation Learning

TL;DR

This paper introduces a parameter-efficient multimodal Transformer architecture for video representation learning, significantly reducing memory requirements and enabling end-to-end training from scratch on large-scale video datasets.

Contribution

It proposes a novel parameter sharing scheme based on low-rank approximation and a modality decomposition, allowing up to 97% parameter reduction and end-to-end training.

Findings

Achieves up to 97% reduction in Transformer parameters.

Enables training from scratch on large-scale video data.

Improves performance on audio-visual classification tasks.

Abstract

The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the parameters of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters of the Transformers up to 97$\%$, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns together with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips (480 frames) from Kinetics-700 and transfer it to audio-visual classification tasks.