Bridging Annotation Gaps: Transferring Labels to Align Object Detection Datasets

TL;DR

This paper introduces Label-Aligned Transfer (LAT), a novel framework for transferring annotations across heterogeneous object detection datasets, improving generalization without manual relabeling or shared label taxonomies.

Contribution

LAT systematically projects annotations into a target dataset's label space using pseudo-labels, a privileged proposal generator, and semantic feature fusion, addressing class and spatial inconsistencies.

Findings

Achieves up to +4.8AP improvement over semi-supervised baselines.

Effectively aligns heterogeneous datasets for better detection performance.

Preserves dataset-specific annotation granularity.

Abstract

Combining multiple object detection datasets offers a path to improved generalisation but is hindered by inconsistencies in class semantics and bounding box annotations. Some methods to address this assume shared label taxonomies and address only spatial inconsistencies; others require manual relabelling, or produce a unified label space, which may be unsuitable when a fixed target label space is required. We propose Label-Aligned Transfer (LAT), a label transfer framework that systematically projects annotations from diverse source datasets into the label space of a target dataset. LAT begins by training dataset-specific detectors to generate pseudo-labels, which are then combined with ground-truth annotations via a Privileged Proposal Generator (PPG) that replaces the region proposal network in two-stage detectors. To further refine region features, a Semantic Feature Fusion (SFF) module injects class-aware context and features from overlapping proposals using a confidence-weighted attention mechanism. This pipeline preserves dataset-specific annotation granularity while enabling many-to-one label space transfer across heterogeneous datasets, resulting in a semantically and spatially aligned representation suitable for training a downstream detector. LAT thus jointly addresses both class-level misalignments and bounding box inconsistencies without relying on shared label spaces or manual annotations. Across multiple benchmarks, LAT demonstrates consistent improvements in target-domain detection performance, achieving gains of up to +4.8AP over semi-supervised baselines.