GHR-VQA: Graph-guided Hierarchical Relational Reasoning for Video Question Answering

TL;DR

GHR-VQA introduces a graph-based hierarchical reasoning framework for video question answering that models human-object interactions across frames, improving interpretability and reasoning accuracy.

Contribution

The paper presents a novel scene graph-based hierarchical reasoning approach for Video QA, emphasizing human-centric interactions and cross-frame reasoning.

Findings

Achieves 7.3% improvement in object-relation reasoning on AGQA dataset.

Effectively models human-object interactions with scene graphs across video frames.

Enhances interpretability through explicit human-rooted structure.

Abstract

We propose GHR-VQA, Graph-guided Hierarchical Relational Reasoning for Video Question Answering (Video QA), a novel human-centric framework that incorporates scene graphs to capture intricate human-object interactions within video sequences. Unlike traditional pixel-based methods, each frame is represented as a scene graph and human nodes across frames are linked to a global root, forming the video-level graph and enabling cross-frame reasoning centered on human actors. The video-level graphs are then processed by Graph Neural Networks (GNNs), transforming them into rich, context-aware embeddings for efficient processing. Finally, these embeddings are integrated with question features in a hierarchical network operating across different abstraction levels, enhancing both local and global understanding of video content. This explicit human-rooted structure enhances interpretability by decomposing actions into human-object interactions and enables a more profound understanding of spatiotemporal dynamics. We validate our approach on the Action Genome Question Answering (AGQA) dataset, achieving significant performance improvements, including a 7.3% improvement in object-relation reasoning over the state of the art.