AdaMixer: A Fast-Converging Query-Based Object Detector

TL;DR

AdaMixer is a novel query-based object detector that improves convergence speed and performance by adaptively sampling features and dynamically decoding them with an adaptive MLP-Mixer, eliminating the need for complex extra networks.

Contribution

It introduces a simple, fast-converging query-based detection architecture with adaptive feature sampling and decoding, enhancing efficiency and accuracy over prior methods.

Findings

Achieves 45.0 AP with 12 training epochs on MS COCO

Reaches 49.5 AP with longer training and advanced backbones

Demonstrates fast convergence and high accuracy without dense attention modules

Abstract

Traditional object detectors employ the dense paradigm of scanning over locations and scales in an image. The recent query-based object detectors break this convention by decoding image features with a set of learnable queries. However, this paradigm still suffers from slow convergence, limited performance, and design complexity of extra networks between backbone and decoder. In this paper, we find that the key to these issues is the adaptability of decoders for casting queries to varying objects. Accordingly, we propose a fast-converging query-based detector, named AdaMixer, by improving the adaptability of query-based decoding processes in two aspects. First, each query adaptively samples features over space and scales based on estimated offsets, which allows AdaMixer to efficiently attend to the coherent regions of objects. Then, we dynamically decode these sampled features with an adaptive MLP-Mixer under the guidance of each query. Thanks to these two critical designs, AdaMixer enjoys architectural simplicity without requiring dense attentional encoders or explicit pyramid networks. On the challenging MS COCO benchmark, AdaMixer with ResNet-50 as the backbone, with 12 training epochs, reaches up to 45.0 AP on the validation set along with 27.9 APs in detecting small objects. With the longer training scheme, AdaMixer with ResNeXt-101-DCN and Swin-S reaches 49.5 and 51.3 AP. Our work sheds light on a simple, accurate, and fast converging architecture for query-based object detectors. The code is made available at https://github.com/MCG-NJU/AdaMixer