Is Heuristic Sampling Necessary in Training Deep Object Detectors?

TL;DR

This paper demonstrates that heuristic sampling is not necessary for training deep object detectors and introduces a simple, hyperparameter-free Sampling-Free mechanism that improves accuracy by addressing classification gradient imbalance.

Contribution

The paper reveals that classification gradient imbalance, not sampling, causes accuracy degradation and proposes a hyperparameter-free method to improve training without heuristic sampling.

Findings

Sampling-Free method outperforms heuristic sampling on COCO and PASCAL VOC.

The approach achieves higher detection accuracy without hyperparameter tuning.

Addressing gradient imbalance is key to training accurate detectors.

Abstract

To train accurate deep object detectors under the extreme foreground-background imbalance, heuristic sampling methods are always necessary, which either re-sample a subset of all training samples (hard sampling methods, \eg biased sampling, OHEM), or use all training samples but re-weight them discriminatively (soft sampling methods, \eg Focal Loss, GHM). In this paper, we challenge the necessity of such hard/soft sampling methods for training accurate deep object detectors. While previous studies have shown that training detectors without heuristic sampling methods would significantly degrade accuracy, we reveal that this degradation comes from an unreasonable classification gradient magnitude caused by the imbalance, rather than a lack of re-sampling/re-weighting. Motivated by our discovery, we propose a simple yet effective \emph{Sampling-Free} mechanism to achieve a reasonable classification gradient magnitude by initialization and loss scaling. Unlike heuristic sampling methods with multiple hyperparameters, our Sampling-Free mechanism is fully data diagnostic, without laborious hyperparameters searching. We verify the effectiveness of our method in training anchor-based and anchor-free object detectors, where our method always achieves higher detection accuracy than heuristic sampling methods on COCO and PASCAL VOC datasets. Our Sampling-Free mechanism provides a new perspective to address the foreground-background imbalance. Our code is released at \url{https://github.com/ChenJoya/sampling-free}.