Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video

TL;DR

This paper introduces EgoViT, a vision Transformer framework that learns stable, persistent object representations from unlabeled egocentric videos by combining intra-frame learning, geometric grounding, and temporal consistency.

Contribution

EgoViT is the first unified Transformer-based approach that jointly discovers and stabilizes object representations in uncurated first-person videos without manual annotations.

Findings

Achieves +8.0% CorLoc in unsupervised object discovery

Improves mIoU by +4.8% in semantic segmentation

Demonstrates robustness to varied geometric priors

Abstract

Humans develop visual intelligence through perceiving and interacting with their environment - a self-supervised learning process grounded in egocentric experience. Inspired by this, we ask how can artificial systems learn stable object representations from continuous, uncurated first-person videos without relying on manual annotations. This setting poses challenges of separating, recognizing, and persistently tracking objects amid clutter, occlusion, and ego-motion. We propose EgoViT, a unified vision Transformer framework designed to learn stable object representations from unlabeled egocentric video. EgoViT bootstraps this learning process by jointly discovering and stabilizing "proto-objects" through three synergistic mechanisms: (1) Proto-object Learning, which uses intra-frame distillation to form discriminative representations; (2) Depth Regularization, which grounds these representations in geometric structure; and (3) Teacher-Filtered Temporal Consistency, which enforces identity over time. This creates a virtuous cycle where initial object hypotheses are progressively refined into stable, persistent representations. The framework is trained end-to-end on unlabeled first-person videos and exhibits robustness to geometric priors of varied origin and quality. On standard benchmarks, EgoViT achieves +8.0% CorLoc improvement in unsupervised object discovery and +4.8% mIoU improvement in semantic segmentation, demonstrating its potential to lay a foundation for robust visual abstraction in embodied intelligence.