Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

TL;DR

This paper introduces an end-to-end multi-granularity contrastive cross-modal collaborative generation model for long-term VideoQA, improving semantic understanding and answer generation by integrating unimodal modeling, contrastive learning, and generative reformulation.

Contribution

The paper proposes a novel end-to-end framework combining joint unimodal modeling, contrastive learning, and generative reformulation for long-term VideoQA, addressing domain-specific reasoning and task formulation issues.

Findings

Outperforms existing methods on six VideoQA datasets.

Effectively captures semantic correspondences through contrastive learning.

Reformulates VideoQA as a generative task for better reasoning.

Abstract

Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.