EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning

TL;DR

EgoIntrospect introduces a comprehensive egocentric dataset with multimodal signals and annotations for understanding users' internal states, enabling new benchmarks for AI assistant research.

Contribution

The paper presents the first egocentric dataset with self-annotations of internal states and benchmarks for multimodal reasoning about user intentions and emotions.

Findings

Existing multimodal models struggle to infer internal states from the dataset.

EgoIntrospect provides 180 hours of synchronized multimodal recordings from 60 subjects.

Benchmarks reveal gaps in current models' ability to understand user internal states.

Abstract

Despite extensive efforts on egocentric video datasets and benchmarks, understanding users' internal states, which is crucial for enabling seamless AI assistant experiences, remains largely overlooked. In this work, we introduce EgoIntrospect, the first egocentric dataset captured in user-driven scenarios with self-annotations that explicitly reveal users' interactive intentions with AI assistants. EgoIntrospect was collected using a cross-device setup, providing synchronized video, audio, gaze, motion, and physiological signals. It consists of 180 hours of recordings from 60 subjects, with an average recording duration of 3 hours per subject. Leveraging EgoIntrospect, we formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory. We further process the annotations to construct benchmarks that evaluate the ability of modern multimodal large language models to reason about users' internal states from egocentric observations. Experiments on our benchmark suggest that existing multimodal large language models struggle to effectively leverage multimodal signals to infer users' subjective internal states. The dataset and annotations will be made publicly available to advance research in egocentric vision and wearable AI assistants. Project page: https://ego-introspect.github.io/