EgoSelf: From Memory to Personalized Egocentric Assistant

TL;DR

EgoSelf is a system that leverages a graph-based memory and a prediction task to personalize egocentric assistants by modeling user behavior and preferences over time.

Contribution

It introduces a novel graph-based memory and a personalized prediction task for integrating long-term user data in egocentric assistants.

Findings

EgoSelf effectively captures temporal and semantic relationships in user data.

The system improves personalization accuracy in egocentric assistant tasks.

Code implementation demonstrates practical applicability.

Abstract

Egocentric assistants often rely on first-person view data to capture user behavior and context for personalized services. Since different users exhibit distinct habits, preferences, and routines, such personalization is essential for truly effective assistance. However, effectively integrating long-term user data for personalization remains a key challenge. To address this, we introduce EgoSelf, a system that includes a graph-based interaction memory constructed from past observations and a dedicated learning task for personalization. The memory captures temporal and semantic relationships among interaction events and entities, from which user-specific profiles are derived. The personalized learning task is formulated as a prediction problem where the model predicts possible future interactions from individual user's historical behavior recorded in the graph. Extensive experiments demonstrate the effectiveness of EgoSelf as a personalized egocentric assistant. Code is available at https://abie-e.github.io/EgoSelf/.