Semantic Enhanced Few-shot Object Detection

TL;DR

This paper introduces a semantic-enhanced few-shot object detection framework that aligns visual features with class name embeddings, uses multimodal fusion, and employs a semantic-aware max-margin loss to improve detection of novel classes with limited data.

Contribution

It proposes a novel fine-tuning framework that leverages semantic embeddings and multimodal fusion to improve few-shot object detection, addressing class confusion issues.

Findings

Outperforms existing methods on Pascal VOC and MS COCO datasets.

Effectively reduces class confusion in low-shot scenarios.

Enhances detection accuracy for novel classes with limited samples.

Abstract

Few-shot object detection~(FSOD), which aims to detect novel objects with limited annotated instances, has made significant progress in recent years. However, existing methods still suffer from biased representations, especially for novel classes in extremely low-shot scenarios. During fine-tuning, a novel class may exploit knowledge from similar base classes to construct its own feature distribution, leading to classification confusion and performance degradation. To address these challenges, we propose a fine-tuning based FSOD framework that utilizes semantic embeddings for better detection. In our proposed method, we align the visual features with class name embeddings and replace the linear classifier with our semantic similarity classifier. Our method trains each region proposal to converge to the corresponding class embedding. Furthermore, we introduce a multimodal feature fusion to augment the vision-language communication, enabling a novel class to draw support explicitly from well-trained similar base classes. To prevent class confusion, we propose a semantic-aware max-margin loss, which adaptively applies a margin beyond similar classes. As a result, our method allows each novel class to construct a compact feature space without being confused with similar base classes. Extensive experiments on Pascal VOC and MS COCO demonstrate the superiority of our method.