From My View to Yours: Ego-to-Exo Transfer in VLMs for Understanding Activities of Daily Living

TL;DR

This paper introduces Ego2ExoVLM, a vision-language model that learns to infer egocentric properties from exocentric videos, enhancing understanding of daily activities without requiring egocentric cameras.

Contribution

It proposes a novel training framework with sequence distillation and adaptive tokens, and introduces Ego-in-Exo Perception, a benchmark for egocentric understanding from exocentric videos.

Findings

Achieves state-of-the-art on ADL-X benchmark.

Outperforms strong baselines on Ego-in-Exo Perception.

Effectively transfers egocentric knowledge to exocentric video understanding.

Abstract

Vision Language Models (VLMs) have achieved strong performance across diverse video understanding tasks. However, their viewpoint invariant training limits their ability to understand egocentric properties (e.g., human object interactions) from exocentric video observations. This limitation is critical for many applications, such as Activities of Daily Living (ADL) monitoring, where the understanding of egocentric properties is essential, and egocentric cameras are impractical to deploy. To address this limitation, we propose Ego2ExoVLM, a VLM that learns to infer egocentric properties from exocentric videos by leveraging time-synchronized ego-exo videos during training. Ego2ExoVLM accomplishes this through the use of two components: Ego2Exo Sequence Distillation, which transfers knowledge from an egocentric teacher to an exocentric student, and Ego Adaptive Visual Tokens, designed to enhance the effectiveness of this knowledge transfer. To measure this capability, we introduce Ego-in-Exo Perception, a benchmark of 3.9K questions curated to explicitly measure the understanding of egocentric properties from exocentric videos. Ego2ExoVLM is evaluated on 10 tasks across Ego-in-Exo Perception and existing ADL benchmarks, achieving state-of-the-art results on the ADL-X benchmark suite and outperforming strong baselines on our proposed benchmark. All code, models, and data will be released at https://github.com/dominickrei/EgoExo4ADL.