Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling

TL;DR

This paper introduces a decoupled spatial-temporal modeling approach for video question answering, combining image- and video-language encoders with a novel temporal pre-training task, leading to superior performance.

Contribution

It proposes a hybrid pipeline with decoupled spatial and temporal encoders and a new pre-training objective for better temporal understanding in video QA.

Findings

Outperforms previous models on video question answering tasks.

Effective decoupling improves spatial and temporal feature learning.

Temporal Referring Modeling enhances temporal relation understanding.

Abstract

While recent large-scale video-language pre-training made great progress in video question answering, the design of spatial modeling of video-language models is less fine-grained than that of image-language models; existing practices of temporal modeling also suffer from weak and noisy alignment between modalities. To learn fine-grained visual understanding, we decouple spatial-temporal modeling and propose a hybrid pipeline, Decoupled Spatial-Temporal Encoders, integrating an image- and a video-language encoder. The former encodes spatial semantics from larger but sparsely sampled frames independently of time, while the latter models temporal dynamics at lower spatial but higher temporal resolution. To help the video-language model learn temporal relations for video QA, we propose a novel pre-training objective, Temporal Referring Modeling, which requires the model to identify temporal positions of events in video sequences. Extensive experiments demonstrate that our model outperforms previous work pre-trained on orders of magnitude larger datasets.