Better Matching, Less Forgetting: A Quality-Guided Matcher for Transformer-based Incremental Object Detection

TL;DR

This paper introduces a novel matcher for transformer-based incremental object detection that reduces forgetting by avoiding background foregrounding errors, leading to improved continual learning performance.

Contribution

It proposes the Q-MCMF matcher, which prunes implausible matches based on geometric quality to prevent background foregrounding, enhancing incremental detection accuracy.

Findings

Outperforms state-of-the-art methods on COCO dataset

Effectively mitigates catastrophic forgetting in incremental detection

Improves foreground-background discrimination in transformer detectors

Abstract

Incremental Object Detection (IOD) aims to continuously learn new object classes without forgetting previously learned ones. A persistent challenge is catastrophic forgetting, primarily attributed to background shift in conventional detectors. While pseudo-labeling mitigates this in dense detectors, we identify a novel, distinct source of forgetting specific to DETR-like architectures: background foregrounding. This arises from the exhaustiveness constraint of the Hungarian matcher, which forcibly assigns every ground truth target to one prediction, even when predictions primarily cover background regions (i.e., low IoU). This erroneous supervision compels the model to misclassify background features as specific foreground classes, disrupting learned representations and accelerating forgetting. To address this, we propose a Quality-guided Min-Cost Max-Flow (Q-MCMF) matcher. To avoid forced assignments, Q-MCMF builds a flow graph and prunes implausible matches based on geometric quality. It then optimizes for the final matching that minimizes cost and maximizes valid assignments. This strategy eliminates harmful supervision from background foregrounding while maximizing foreground learning signals. Extensive experiments on the COCO dataset under various incremental settings demonstrate that our method consistently outperforms existing state-of-the-art approaches.