ScaleDet: A Scalable Multi-Dataset Object Detector

TL;DR

ScaleDet is a scalable multi-dataset object detector that unifies labels across datasets through visual-textual alignment, enabling improved generalization and performance on diverse datasets with seen and unseen classes.

Contribution

We introduce ScaleDet, a scalable multi-dataset detector that uses a simple semantic label space and visual-textual alignment to unify labels without manual relabeling or complex optimization.

Findings

Achieves high mAP scores on multiple datasets, e.g., 50.7 on LVIS and 58.8 on COCO.

Surpasses state-of-the-art detectors with the same backbone architecture.

Generalizes well to unseen classes and diverse datasets.

Abstract

Multi-dataset training provides a viable solution for exploiting heterogeneous large-scale datasets without extra annotation cost. In this work, we propose a scalable multi-dataset detector (ScaleDet) that can scale up its generalization across datasets when increasing the number of training datasets. Unlike existing multi-dataset learners that mostly rely on manual relabelling efforts or sophisticated optimizations to unify labels across datasets, we introduce a simple yet scalable formulation to derive a unified semantic label space for multi-dataset training. ScaleDet is trained by visual-textual alignment to learn the label assignment with label semantic similarities across datasets. Once trained, ScaleDet can generalize well on any given upstream and downstream datasets with seen and unseen classes. We conduct extensive experiments using LVIS, COCO, Objects365, OpenImages as upstream datasets, and 13 datasets from Object Detection in the Wild (ODinW) as downstream datasets. Our results show that ScaleDet achieves compelling strong model performance with an mAP of 50.7 on LVIS, 58.8 on COCO, 46.8 on Objects365, 76.2 on OpenImages, and 71.8 on ODinW, surpassing state-of-the-art detectors with the same backbone.