Ego-Only: Egocentric Action Detection without Exocentric Transferring

TL;DR

Ego-Only introduces a novel egocentric action detection method that trains models from scratch without relying on exocentric transferring, achieving state-of-the-art results on multiple datasets.

Contribution

The paper demonstrates that effective egocentric action detection can be achieved without exocentric transferring by using a masked autoencoder for feature learning.

Findings

Ego-Only outperforms previous exocentric transferring methods.

It achieves state-of-the-art results on Ego4D, EPIC-Kitchens-100, and Charades-Ego.

The approach requires no exocentric data or transferring.

Abstract

We present Ego-Only, the first approach that enables state-of-the-art action detection on egocentric (first-person) videos without any form of exocentric (third-person) transferring. Despite the content and appearance gap separating the two domains, large-scale exocentric transferring has been the default choice for egocentric action detection. This is because prior works found that egocentric models are difficult to train from scratch and that transferring from exocentric representations leads to improved accuracy. However, in this paper, we revisit this common belief. Motivated by the large gap separating the two domains, we propose a strategy that enables effective training of egocentric models without exocentric transferring. Our Ego-Only approach is simple. It trains the video representation with a masked autoencoder finetuned for temporal segmentation. The learned features are then fed to an off-the-shelf temporal action localization method to detect actions. We find that this renders exocentric transferring unnecessary by showing remarkably strong results achieved by this simple Ego-Only approach on three established egocentric video datasets: Ego4D, EPIC-Kitchens-100, and Charades-Ego. On both action detection and action recognition, Ego-Only outperforms previous best exocentric transferring methods that use orders of magnitude more labels. Ego-Only sets new state-of-the-art results on these datasets and benchmarks without exocentric data.