TeleEgo: Benchmarking Egocentric AI Assistants in the Wild

TL;DR

TeleEgo is a comprehensive benchmark for evaluating egocentric AI assistants in real-world, streaming scenarios, focusing on multi-modal understanding, memory, and responsiveness over long durations.

Contribution

It introduces TeleEgo, a long-duration, multi-modal benchmark with new metrics for assessing real-time accuracy and long-term memory in egocentric AI assistants.

Findings

Current models show limited real-time accuracy in streaming settings.

The benchmark reveals challenges in long-term memory retention.

TeleEgo provides a platform for systematic evaluation of egocentric AI capabilities.

Abstract

Egocentric AI assistants in real-world settings must process multi-modal inputs (video, audio, text), respond in real time, and retain evolving long-term memory. However, existing benchmarks typically evaluate these abilities in isolation, lack realistic streaming scenarios, or support only short-term tasks. We introduce \textbf{TeleEgo}, a long-duration, streaming, omni-modal benchmark for evaluating egocentric AI assistants in realistic daily contexts. The dataset features over 14 hours per participant of synchronized egocentric video, audio, and text across four domains: work \& study, lifestyle \& routines, social activities, and outings \& culture. All data is aligned on a unified global timeline and includes high-quality visual narrations and speech transcripts, curated through human refinement.TeleEgo defines 12 diagnostic subtasks across three core capabilities: Memory (recalling past events), Understanding (interpreting the current moment), and Cross-Memory Reasoning (linking distant events). It contains 3,291 human-verified QA items spanning multiple question formats (single-choice, binary, multi-choice, and open-ended), evaluated strictly in a streaming setting. We propose Real-Time Accuracy (RTA) to jointly capture correctness and responsiveness under tight decision windows, and Memory Persistence Time (MPT) as a forward-looking metric for long-term retention in continuous streams. In this work, we report RTA results for current models and release TeleEgo, together with an MPT evaluation framework, as a realistic and extensible benchmark for future egocentric assistants with stronger streaming memory, enabling systematic study of both real-time behavior and long-horizon memory.