EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs

TL;DR

EASG-Bench is a new video question-answering benchmark focusing on egocentric videos with dynamic scene graphs, revealing challenges in temporal understanding for current language and video models.

Contribution

We introduce EASG-Bench, a novel benchmark for egocentric video Q&A based on dynamic scene graphs, and evaluate models to identify gaps in temporal reasoning.

Findings

Significant performance gap in temporal questions for current models.

Language-only models underperform compared to video-LLMs.

Benchmark and code are publicly available for further research.

Abstract

We introduce EASG-Bench, a question-answering benchmark for egocentric videos where the question-answering pairs are created from spatio-temporally grounded dynamic scene graphs capturing intricate relationships among actors, actions, and objects. We propose a systematic evaluation framework and evaluate several language-only and video large language models (video-LLMs) on this benchmark. We observe a performance gap in language-only and video-LLMs, especially on questions focusing on temporal ordering, thus identifying a research gap in the area of long-context video understanding. To promote the reproducibility of our findings and facilitate further research, the benchmark and accompanying code are available at the following GitHub page: https://github.com/fpv-iplab/EASG-bench.