MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering

TL;DR

MIST is a novel multi-modal transformer model designed for long-form VideoQA, efficiently selecting relevant frames and regions for multi-event reasoning, achieving state-of-the-art results with improved interpretability.

Contribution

The paper introduces MIST, a new model that adaptively selects relevant spatial-temporal segments for long-form VideoQA, addressing computational challenges and enhancing reasoning capabilities.

Findings

Achieves state-of-the-art performance on four VideoQA datasets.

Demonstrates superior computational efficiency compared to dense sampling methods.

Provides improved interpretability through selective attention mechanisms.

Abstract

To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must. Existing multi-modal VQA models achieve promising performance on images or short video clips, especially with the recent success of large-scale multi-modal pre-training. However, when extending these methods to long-form videos, new challenges arise. On the one hand, using a dense video sampling strategy is computationally prohibitive. On the other hand, methods relying on sparse sampling struggle in scenarios where multi-event and multi-granularity visual reasoning are required. In this work, we introduce a new model named Multi-modal Iterative Spatial-temporal Transformer (MIST) to better adapt pre-trained models for long-form VideoQA. Specifically, MIST decomposes traditional dense spatial-temporal self-attention into cascaded segment and region selection modules that adaptively select frames and image regions that are closely relevant to the question itself. Visual concepts at different granularities are then processed efficiently through an attention module. In addition, MIST iteratively conducts selection and attention over multiple layers to support reasoning over multiple events. The experimental results on four VideoQA datasets, including AGQA, NExT-QA, STAR, and Env-QA, show that MIST achieves state-of-the-art performance and is superior at computation efficiency and interpretability.