Balanced Hierarchical Contrastive Learning with Decoupled Queries for Fine-grained Object Detection in Remote Sensing Images

TL;DR

This paper introduces a balanced hierarchical contrastive loss with decoupled queries within the DETR framework to improve fine-grained object detection in remote sensing images, effectively handling data imbalance and semantic hierarchy issues.

Contribution

It proposes a novel loss and learning strategy that balance class contributions and decouple classification and localization tasks in transformer-based detection models.

Findings

Outperforms state-of-the-art methods on three hierarchical remote sensing datasets.

Effectively handles class imbalance across hierarchical levels.

Enhances fine-grained detection accuracy in remote sensing images.

Abstract

Fine-grained remote sensing datasets often use hierarchical label structures to differentiate objects in a coarse-to-fine manner, with each object annotated across multiple levels. However, embedding this semantic hierarchy into the representation learning space to improve fine-grained detection performance remains challenging. Previous studies have applied supervised contrastive learning at different hierarchical levels to group objects under the same parent class while distinguishing sibling subcategories. Nevertheless, they overlook two critical issues: (1) imbalanced data distribution across the label hierarchy causes high-frequency classes to dominate the learning process, and (2) learning semantic relationships among categories interferes with class-agnostic localization. To address these issues, we propose a balanced hierarchical contrastive loss combined with a decoupled learning strategy within the detection transformer (DETR) framework. The proposed loss introduces learnable class prototypes and equilibrates gradients contributed by different classes at each hierarchical level, ensuring that each hierarchical class contributes equally to the loss computation in every mini-batch. The decoupled strategy separates DETR's object queries into classification and localization sets, enabling task-specific feature extraction and optimization. Experiments on three fine-grained datasets with hierarchical annotations demonstrate that our method outperforms state-of-the-art approaches.