Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos

TL;DR

This paper presents Ego-Exo, a method that leverages large-scale third-person videos to improve egocentric video models by discovering predictive signals and using knowledge distillation, leading to state-of-the-art results.

Contribution

The paper introduces a novel framework that transfers knowledge from third-person to egocentric videos, enhancing model performance by combining scale, diversity, and egocentric-specific features.

Findings

Outperforms all baselines on egocentric activity recognition tasks.

Achieves state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.

Seamlessly integrates with standard video models.

Abstract

We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our Ego-Exo framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.