Ego4OOD: Rethinking Egocentric Video Domain Generalization via Covariate Shift Scoring

TL;DR

This paper introduces Ego4OOD, a new benchmark for egocentric video domain generalization focusing on covariate shifts, and proposes a binary classification approach that improves out-of-distribution recognition performance.

Contribution

The paper presents Ego4OOD, a benchmark emphasizing covariate diversity and a binary training method that enhances domain generalization in egocentric videos.

Findings

A covariate shift metric correlates with recognition performance.

A simple two-layer network achieves competitive results.

Ego4OOD reduces concept shift by using semantically coherent categories.

Abstract

Egocentric video action recognition under domain shifts remains challenging due to large intra-class spatio-temporal variability, long-tailed feature distributions, and strong correlations between actions and environments. Existing benchmarks for egocentric domain generalization often conflate covariate shifts with concept shifts, making it difficult to reliably evaluate a model's ability to generalize across input distributions. To address this limitation, we introduce Ego4OOD, a domain generalization benchmark derived from Ego4D that emphasizes measurable covariate diversity while reducing concept shift through semantically coherent, moment-level action categories. Ego4OOD spans eight geographically distinct domains and is accompanied by a clustering-based covariate shift metric that provides a quantitative proxy for domain difficulty. We further leverage a one-vs-all binary training objective that decomposes multi-class action recognition into independent binary classification tasks. This formulation is particularly well-suited for covariate shift by reducing interference between visually similar classes under feature distribution shift. Using this formulation, we show that a lightweight two-layer fully connected network achieves performance competitive with state-of-the-art egocentric domain generalization methods on both Argo1M and Ego4OOD, despite using fewer parameters and no additional modalities. Our empirical analysis demonstrates a clear relationship between measured covariate shift and recognition performance, highlighting the importance of controlled benchmarks and quantitative domain characterization for studying out-of-distribution generalization in egocentric video.