Unique Lives, Shared World: Learning from Single-Life Videos

TL;DR

This paper introduces a new single-life learning paradigm where models trained on egocentric videos from one individual develop aligned, generalizable visual representations that transfer well to other tasks and environments, even with limited data.

Contribution

The paper proposes the single-life learning paradigm, demonstrating that models trained on individual egocentric videos develop aligned, transferable visual representations with minimal data, a novel approach in self-supervised vision learning.

Findings

Models trained on different lives develop highly aligned geometric understanding.

Single-life models effectively transfer to downstream tasks like depth estimation.

Training on 30 hours of one person's life matches performance of 30 hours of web data.

Abstract

We introduce the "single-life" learning paradigm, where we train a distinct vision model exclusively on egocentric videos captured by one individual. We leverage the multiple viewpoints naturally captured within a single life to learn a visual encoder in a self-supervised manner. Our experiments demonstrate three key findings. First, models trained independently on different lives develop a highly aligned geometric understanding. We demonstrate this by training visual encoders on distinct datasets each capturing a different life, both indoors and outdoors, as well as introducing a novel cross-attention-based metric to quantify the functional alignment of the internal representations developed by different models. Second, we show that single-life models learn generalizable geometric representations that effectively transfer to downstream tasks, such as depth estimation, in unseen environments. Third, we demonstrate that training on up to 30 hours from one week of the same person's life leads to comparable performance to training on 30 hours of diverse web data, highlighting the strength of single-life representation learning. Overall, our results establish that the shared structure of the world, both leads to consistency in models trained on individual lives, and provides a powerful signal for visual representation learning.