AoE: Always-on Egocentric Human Video Collection for Embodied AI

TL;DR

This paper introduces AoE, a low-cost, scalable system for collecting egocentric human video data using smartphones, which enhances embodied AI models by providing high-quality real-world interaction data.

Contribution

The paper presents a novel, scalable data collection system leveraging smartphones and cloud processing to gather egocentric videos from humans worldwide, reducing costs and hardware dependencies.

Findings

High-quality egocentric data improves real-world generalization.

AoE system enables large-scale, distributed data collection.

Automated labeling enhances data processing efficiency.

Abstract

Embodied foundation models require large-scale, high-quality real-world interaction data for pre-training and scaling. However, existing data collection methods suffer from high infrastructure costs, complex hardware dependencies, and limited interaction scope, making scalable expansion challenging. In fact, humans themselves are ideal physically embodied agents. Therefore, obtaining egocentric real-world interaction data from globally distributed "human agents" offers advantages of low cost and sustainability. To this end, we propose the Always-on Egocentric (AoE) data collection system, which aims to simplify hardware dependencies by leveraging humans themselves and their smartphones, enabling low-cost, highly efficient, and scene-agnostic real-world interaction data collection to address the challenge of data scarcity. Specifically, we first employ an ergonomic neck-mounted smartphone holder to enable low-barrier, large-scale egocentric data collection through a cloud-edge collaborative architecture. Second, we develop a cross-platform mobile APP that leverages on-device compute for real-time processing, while the cloud hosts automated labeling and filtering pipelines that transform raw videos into high-quality training data. Finally, the AoE system supports distributed Ego video data collection by anyone, anytime, and anywhere. We evaluate AoE on data preprocessing quality and downstream tasks, demonstrating that high-quality egocentric data significantly boosts real-world generalization.