EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams

TL;DR

EgoPro-Bench is a new benchmark for training and evaluating proactive, personalized interaction capabilities in egocentric video streams, emphasizing timing and user context.

Contribution

It introduces a comprehensive dataset, evaluation protocol, and a novel interaction principle to advance proactive multimodal large language models.

Findings

EgoPro-Bench improves intention understanding in MLLMs.

Models trained on EgoPro-Bench accurately identify HMI timing.

The benchmark enables development of more user-centric proactive agents.

Abstract

Existing Multimodal Large Language Models (MLLMs) remain primarily reactive, failing to continuously perceive environments or proactively assist users. While emerging benchmarks address proactivity, they are largely confined to alert scenarios, neglect personalized context, and fail to evaluate the precise timing of human-machine interactions (HMI).In this paper, we introduce EgoPro-Bench, a novel benchmark for training and evaluating proactive interaction capabilities based on streaming egocentric videos; it comprises 2,400 videos in the evaluation set and over 12,000 videos in the training set.Unlike previous works, EgoPro-Bench leverages simulated user profiles to generate diverse user intentions and to construct high-fidelity HMI data across 12 distinct domains.Subsequently, we propose a specialized evaluation protocol and metrics, train proactive interaction models designed for efficient reasoning and low-latency interaction on streaming video data, and conduct comprehensive evaluations.Furthermore, we introduce an interaction principle termed "short thinking, better interaction", which allocates a limited token budget prior to intent recognition, thereby enhancing interaction performance.The experiments demonstrate that EgoPro-Bench substantially enhances the intention understanding capabilities of MLLMs and enables accurate identification of appropriate timings for HMI, thereby laying a solid foundation for next-generation user-centric proactive interactive agents.