Wanderlust: Online Continual Object Detection in the Real World

TL;DR

This paper introduces OAK, a new egocentric video dataset and benchmark for online continual object detection, emphasizing realistic, long-term, and dynamic environment scenarios to advance continual learning research.

Contribution

It provides a novel, realistic dataset and benchmark for online continual object detection, including new evaluation metrics and baseline studies.

Findings

Benchmark captures natural distribution shifts in egocentric videos.

Baseline studies demonstrate challenges in catastrophic forgetting.

New metrics effectively evaluate continual learning performance.

Abstract

Online continual learning from data streams in dynamic environments is a critical direction in the computer vision field. However, realistic benchmarks and fundamental studies in this line are still missing. To bridge the gap, we present a new online continual object detection benchmark with an egocentric video dataset, Objects Around Krishna (OAK). OAK adopts the KrishnaCAM videos, an ego-centric video stream collected over nine months by a graduate student. OAK provides exhaustive bounding box annotations of 80 video snippets (~17.5 hours) for 105 object categories in outdoor scenes. The emergence of new object categories in our benchmark follows a pattern similar to what a single person might see in their day-to-day life. The dataset also captures the natural distribution shifts as the person travels to different places. These egocentric long-running videos provide a realistic playground for continual learning algorithms, especially in online embodied settings. We also introduce new evaluation metrics to evaluate the model performance and catastrophic forgetting and provide baseline studies for online continual object detection. We believe this benchmark will pose new exciting challenges for learning from non-stationary data in continual learning. The OAK dataset and the associated benchmark are released at https://oakdata.github.io/.