Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoning

TL;DR

This paper introduces I-VQA, a new task and dataset for answering questions about videos where explicit visual evidence is unavailable, and proposes IRM, a dual-clue reasoning framework that outperforms existing models.

Contribution

The paper presents the first implicit VideoQA task and dataset, along with a novel IRM framework that models dual clues for reasoning in challenging scenarios.

Findings

IRM outperforms GPT-4o, OpenAI-o3, and fine-tuned VideoChat2.

IRM achieves state-of-the-art results on implicit advertisement understanding.

IRM performs well on future prediction in traffic-VQA.

Abstract

Video Question Answering (VideoQA) aims to answer natural language questions based on the given video, with prior work primarily focusing on identifying the duration of relevant segments, referred to as explicit visual evidence. However, explicit visual evidence is not always directly available, particularly when questions target symbolic meanings or deeper intentions, leading to significant performance degradation. To fill this gap, we introduce a novel task and dataset, $\textbf{I}$mplicit $\textbf{V}$ideo $\textbf{Q}$uestion $\textbf{A}$nswering (I-VQA), which focuses on answering questions in scenarios where explicit visual evidence is inaccessible. Given an implicit question and its corresponding video, I-VQA requires answering based on the contextual visual cues present within the video. To tackle I-VQA, we propose a novel reasoning framework, IRM (Implicit Reasoning Model), incorporating dual-stream modeling of contextual actions and intent clues as implicit reasoning chains. IRM comprises the Action-Intent Module (AIM) and the Visual Enhancement Module (VEM). AIM deduces and preserves question-related dual clues by generating clue candidates and performing relation deduction. VEM enhances contextual visual representation by leveraging key contextual clues. Extensive experiments validate the effectiveness of our IRM in I-VQA tasks, outperforming GPT-4o, OpenAI-o3, and fine-tuned VideoChat2 by $0.76\%$, $1.37\%$, and $4.87\%$, respectively. Additionally, IRM performs SOTA on similar implicit advertisement understanding and future prediction in traffic-VQA. Datasets and codes are available for double-blind review in anonymous repo: https://github.com/tychen-SJTU/Implicit-VideoQA.