Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception

TL;DR

This paper introduces IMP, a scalable multimodal perception model that combines alternating gradient descent and mixture-of-experts to efficiently handle diverse modalities and tasks, achieving state-of-the-art results in zero-shot video classification.

Contribution

The paper proposes a novel integrated multimodal perception framework using AGD and MoE, enabling efficient scaling and improved performance across multiple modalities and tasks.

Findings

AGD on diverse modalities enhances model performance.

MoE significantly improves accuracy and mitigates modality conflicts.

Sparse IMP-MoE-L achieves state-of-the-art zero-shot video classification with reduced training cost.

Abstract

We present Integrated Multimodal Perception (IMP), a simple and scalable multimodal multi-task training and modeling approach. IMP integrates multimodal inputs including image, video, text, and audio into a single Transformer encoder with minimal modality-specific components. IMP makes use of a novel design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts (MoE) for efficient model and task scaling. We conduct extensive empirical studies and reveal the following key insights: 1) Performing gradient descent updates by alternating on diverse modalities, loss functions, and tasks, with varying input resolutions, efficiently improves the model. 2) Sparsification with MoE on a single modality-agnostic encoder substantially improves the performance, outperforming dense models that use modality-specific encoders or additional fusion layers and greatly mitigates the conflicts between modalities. IMP achieves competitive performance on a wide range of downstream tasks including video classification, image classification, image-text, and video-text retrieval. Most notably, we train a sparse IMP-MoE-L variant focusing on video tasks that achieves new state-of-the-art in zero-shot video classification: 77.0% on Kinetics-400, 76.8% on Kinetics-600, and 68.3% on Kinetics-700, improving the previous state-of-the-art by +5%, +6.7%, and +5.8%, respectively, while using only 15% of their total training computational cost.