Cross-domain Few-shot Object Detection with Multi-modal Textual Enrichment

TL;DR

This paper introduces a meta-learning framework that leverages rich textual semantics to improve cross-domain few-shot object detection, effectively addressing domain shift issues by integrating visual and linguistic features.

Contribution

The paper proposes a novel multi-modal architecture with feature aggregation and semantic rectification modules for better domain adaptation in few-shot object detection.

Findings

Significantly outperforms existing few-shot detection methods on benchmarks.

Effective alignment of visual and linguistic features across domains.

Enhanced understanding of language improves detection accuracy.

Abstract

Advancements in cross-modal feature extraction and integration have significantly enhanced performance in few-shot learning tasks. However, current multi-modal object detection (MM-OD) methods often experience notable performance degradation when encountering substantial domain shifts. We propose that incorporating rich textual information can enable the model to establish a more robust knowledge relationship between visual instances and their corresponding language descriptions, thereby mitigating the challenges of domain shift. Specifically, we focus on the problem of Cross-Domain Multi-Modal Few-Shot Object Detection (CDMM-FSOD) and introduce a meta-learning-based framework designed to leverage rich textual semantics as an auxiliary modality to achieve effective domain adaptation. Our new architecture incorporates two key components: (i) A multi-modal feature aggregation module, which aligns visual and linguistic feature embeddings to ensure cohesive integration across modalities. (ii) A rich text semantic rectification module, which employs bidirectional text feature generation to refine multi-modal feature alignment, thereby enhancing understanding of language and its application in object detection. We evaluate the proposed method on common cross-domain object detection benchmarks and demonstrate that it significantly surpasses existing few-shot object detection approaches.