ASAG: Building Strong One-Decoder-Layer Sparse Detectors via Adaptive Sparse Anchor Generation

TL;DR

This paper introduces ASAG, an adaptive sparse anchor generator that dynamically predicts image-specific anchors, significantly improving the performance of one-decoder-layer sparse detectors while maintaining fast inference speeds.

Contribution

The paper proposes a novel adaptive anchor generation method that alleviates feature conflict, enabling one-decoder-layer detectors to close the performance gap with multi-layer detectors.

Findings

ASAG outperforms dense-initialized detectors in accuracy.

The method achieves a better speed-accuracy trade-off.

Extensive experiments validate the effectiveness of ASAG.

Abstract

Recent sparse detectors with multiple, e.g. six, decoder layers achieve promising performance but much inference time due to complex heads. Previous works have explored using dense priors as initialization and built one-decoder-layer detectors. Although they gain remarkable acceleration, their performance still lags behind their six-decoder-layer counterparts by a large margin. In this work, we aim to bridge this performance gap while retaining fast speed. We find that the architecture discrepancy between dense and sparse detectors leads to feature conflict, hampering the performance of one-decoder-layer detectors. Thus we propose Adaptive Sparse Anchor Generator (ASAG) which predicts dynamic anchors on patches rather than grids in a sparse way so that it alleviates the feature conflict problem. For each image, ASAG dynamically selects which feature maps and which locations to predict, forming a fully adaptive way to generate image-specific anchors. Further, a simple and effective Query Weighting method eases the training instability from adaptiveness. Extensive experiments show that our method outperforms dense-initialized ones and achieves a better speed-accuracy trade-off. The code is available at \url{https://github.com/iSEE-Laboratory/ASAG}.