Efficient End-to-End Video Question Answering with Pyramidal Multimodal Transformer

TL;DR

This paper introduces a pyramidal multimodal transformer for end-to-end VideoQA that efficiently models multi-scale video-language interactions without relying on large pre-trained feature extractors, achieving competitive results.

Contribution

The paper proposes a novel pyramidal multimodal transformer with anisotropic pyramid structures and scale-specific interactions for efficient VideoQA.

Findings

Achieves comparable or better performance than state-of-the-art methods.

Demonstrates high computational efficiency and scalability.

Effective in text-to-video retrieval tasks.

Abstract

This paper presents a new method for end-to-end Video Question Answering (VideoQA), aside from the current popularity of using large-scale pre-training with huge feature extractors. We achieve this with a pyramidal multimodal transformer (PMT) model, which simply incorporates a learnable word embedding layer, a few convolutional and transformer layers. We use the anisotropic pyramid to fulfill video-language interactions across different spatio-temporal scales. In addition to the canonical pyramid, which includes both bottom-up and top-down pathways with lateral connections, novel strategies are proposed to decompose the visual feature stream into spatial and temporal sub-streams at different scales and implement their interactions with the linguistic semantics while preserving the integrity of local and global semantics. We demonstrate better or on-par performances with high computational efficiency against state-of-the-art methods on five VideoQA benchmarks. Our ablation study shows the scalability of our model that achieves competitive results for text-to-video retrieval by leveraging feature extractors with reusable pre-trained weights, and also the effectiveness of the pyramid.