ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

TL;DR

ProactiveVideoQA introduces a new benchmark and metric for evaluating multimodal video systems' ability to proactively interact in real-time, emphasizing temporal response accuracy and user experience.

Contribution

It presents the first comprehensive benchmark and a novel temporal-aware metric, PAUC, for assessing proactive interaction in multimodal video dialogue systems.

Findings

PAUC aligns better with human preferences than traditional metrics.

Baseline systems show room for improvement in proactive interaction capabilities.

The benchmark facilitates future research in proactive multimodal dialogue systems.

Abstract

With the growing research focus on multimodal dialogue systems, the capability for proactive interaction is gradually gaining recognition. As an alternative to conventional turn-by-turn dialogue, users increasingly expect multimodal systems to be more initiative, for example, by autonomously determining the timing of multi-turn responses in real time during video playback. To facilitate progress in this emerging area, we introduce ProactiveVideoQA, the first comprehensive benchmark to evaluate a system's ability to engage in proactive interaction. Since model responses are generated at varying timestamps, we further propose PAUC, the first metric that accounts for the temporal dynamics of model responses. This enables a more accurate evaluation of systems operating in proactive settings. Through extensive benchmarking of various baseline systems on ProactiveVideoQA and a user study of human preferences, we show that PAUC is in better agreement with human preferences than traditional evaluation metrics, which typically only consider the textual content of responses. These findings demonstrate that PAUC provides a more faithful assessment of user experience in proactive interaction scenarios. Project homepage: https://github.com/yellow-binary-tree/ProactiveVideoQA