Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed Detection

TL;DR

This paper introduces a unified resampling strategy combining image-level and object-level approaches to improve long-tailed detection, utilizing an object-centric memory replay to enhance class balance and detection performance.

Contribution

It proposes a novel joint resampling strategy (RIO) that unifies image and object-level resampling for long-tailed detection, with an object-centric memory replay mechanism.

Findings

Outperforms state-of-the-art methods on LVIS v0.5 dataset

Effective object-level resampling with minimal extra computation

Implicit feature augmentation from model updates enhances detection

Abstract

Training on datasets with long-tailed distributions has been challenging for major recognition tasks such as classification and detection. To deal with this challenge, image resampling is typically introduced as a simple but effective approach. However, we observe that long-tailed detection differs from classification since multiple classes may be present in one image. As a result, image resampling alone is not enough to yield a sufficiently balanced distribution at the object level. We address object-level resampling by introducing an object-centric memory replay strategy based on dynamic, episodic memory banks. Our proposed strategy has two benefits: 1) convenient object-level resampling without significant extra computation, and 2) implicit feature-level augmentation from model updates. We show that image-level and object-level resamplings are both important, and thus unify them with a joint resampling strategy (RIO). Our method outperforms state-of-the-art long-tailed detection and segmentation methods on LVIS v0.5 across various backbones. Code is available at https://github.com/NVlabs/RIO.