Large Language Models are Temporal and Causal Reasoners for Video Question Answering

TL;DR

This paper introduces Flipped-VQA, a novel framework that enhances Video Question Answering by encouraging models to understand complex relationships among video, question, and answer, reducing linguistic bias and improving performance across multiple benchmarks.

Contribution

The paper proposes Flipped-VQA, a general framework for LLMs that improves VideoQA by predicting all triplet combinations, effectively leveraging linguistic priors while mitigating bias.

Findings

LLaMA-VQA outperforms existing models on five VideoQA benchmarks.

Flipped-VQA improves performance across various LLMs like OPT and GPT-J.

The framework reduces linguistic bias and enhances understanding of complex video-question-answer relationships.

Abstract

Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks. We observe that the LLMs provide effective priors in exploiting $\textit{linguistic shortcuts}$ for temporal and causal reasoning in Video Question Answering (VideoQA). However, such priors often cause suboptimal results on VideoQA by leading the model to over-rely on questions, $\textit{i.e.}$, $\textit{linguistic bias}$, while ignoring visual content. This is also known as `ungrounded guesses' or `hallucinations'. To address this problem while leveraging LLMs' prior on VideoQA, we propose a novel framework, Flipped-VQA, encouraging the model to predict all the combinations of $\langle$V, Q, A$\rangle$ triplet by flipping the source pair and the target label to understand their complex relationships, $\textit{i.e.}$, predict A, Q, and V given a VQ, VA, and QA pairs, respectively. In this paper, we develop LLaMA-VQA by applying Flipped-VQA to LLaMA, and it outperforms both LLMs-based and non-LLMs-based models on five challenging VideoQA benchmarks. Furthermore, our Flipped-VQA is a general framework that is applicable to various LLMs (OPT and GPT-J) and consistently improves their performances. We empirically demonstrate that Flipped-VQA not only enhances the exploitation of linguistic shortcuts but also mitigates the linguistic bias, which causes incorrect answers over-relying on the question. Code is available at https://github.com/mlvlab/Flipped-VQA.