Redundancy-aware Transformer for Video Question Answering

TL;DR

This paper introduces RaFormer, a transformer architecture for VideoQA that reduces redundancy by focusing on object-level changes and selective vision-language interactions, leading to state-of-the-art results.

Contribution

The paper proposes a novel redundancy-aware transformer that addresses neighboring-frame and cross-modal redundancies in VideoQA tasks, improving performance.

Findings

Achieves state-of-the-art results on multiple VideoQA benchmarks.

Effectively reduces neighboring-frame redundancy by focusing on object-level changes.

Improves vision-language interaction efficiency through adaptive sampling.

Abstract

This paper identifies two kinds of redundancy in the current VideoQA paradigm. Specifically, the current video encoders tend to holistically embed all video clues at different granularities in a hierarchical manner, which inevitably introduces \textit{neighboring-frame redundancy} that can overwhelm detailed visual clues at the object level. Subsequently, prevailing vision-language fusion designs introduce the \textit{cross-modal redundancy} by exhaustively fusing all visual elements with question tokens without explicitly differentiating their pairwise vision-language interactions, thus making a pernicious impact on the answering. To this end, we propose a novel transformer-based architecture, that aims to model VideoQA in a redundancy-aware manner. To address the neighboring-frame redundancy, we introduce a video encoder structure that emphasizes the object-level change in neighboring frames, while adopting an out-of-neighboring message-passing scheme that imposes attention only on distant frames. As for the cross-modal redundancy, we equip our fusion module with a novel adaptive sampling, which explicitly differentiates the vision-language interactions by identifying a small subset of visual elements that exclusively support the answer. Upon these advancements, we find this \underline{R}edundancy-\underline{a}ware trans\underline{former} (RaFormer) can achieve state-of-the-art results on multiple VideoQA benchmarks.