LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

TL;DR

LiVLR is a lightweight, flexible visual-linguistic reasoning framework for VideoQA that effectively integrates multi-modal content at different semantic levels, achieving superior performance on benchmark datasets.

Contribution

The paper introduces LiVLR, a novel lightweight VideoQA framework with a diversity-aware reasoning module for flexible multi-modal content integration.

Findings

Outperforms existing methods on MRSVTT-QA and KnowIT VQA datasets.

Effective multi-grained visual and linguistic representations.

Key components validated through extensive ablation studies.

Abstract

Video Question Answering (VideoQA), aiming to correctly answer the given question based on understanding multi-modal video content, is challenging due to the rich video content. From the perspective of video understanding, a good VideoQA framework needs to understand the video content at different semantic levels and flexibly integrate the diverse video content to distill question-related content. To this end, we propose a Lightweight Visual-Linguistic Reasoning framework named LiVLR. Specifically, LiVLR first utilizes the graph-based Visual and Linguistic Encoders to obtain multi-grained visual and linguistic representations. Subsequently, the obtained representations are integrated with the devised Diversity-aware Visual-Linguistic Reasoning module (DaVL). The DaVL considers the difference between the different types of representations and can flexibly adjust the importance of different types of representations when generating the question-related joint representation, which is an effective and general representation integration method. The proposed LiVLR is lightweight and shows its performance advantage on two VideoQA benchmarks, MRSVTT-QA and KnowIT VQA. Extensive ablation studies demonstrate the effectiveness of LiVLR key components.