CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets

TL;DR

CM3T is a versatile, efficient framework that adapts transformer models for multimodal video classification, requiring minimal retraining and achieving state-of-the-art results across diverse datasets.

Contribution

Introduces CM3T, a model-agnostic plugin architecture with novel adapters for efficient cross-modal learning without extensive retraining.

Findings

Achieves comparable or better results than state-of-the-art with fewer trainable parameters.

Demonstrates effectiveness across multiple datasets and recording settings.

Requires only 12.8% and 22.3% of parameters for video and multimodal processing.

Abstract

Challenges in cross-learning involve inhomogeneous or even inadequate amount of training data and lack of resources for retraining large pretrained models. Inspired by transfer learning techniques in NLP, adapters and prefix tuning, this paper presents a new model-agnostic plugin architecture for cross-learning, called CM3T, that adapts transformer-based models to new or missing information. We introduce two adapter blocks: multi-head vision adapters for transfer learning and cross-attention adapters for multimodal learning. Training becomes substantially efficient as the backbone and other plugins do not need to be finetuned along with these additions. Comparative and ablation studies on three datasets Epic-Kitchens-100, MPIIGroupInteraction and UDIVA v0.5 show efficacy of this framework on different recording settings and tasks. With only 12.8% trainable parameters compared to the backbone to process video input and only 22.3% trainable parameters for two additional modalities, we achieve comparable and even better results than the state-of-the-art. CM3T has no specific requirements for training or pretraining and is a step towards bridging the gap between a general model and specific practical applications of video classification.