Fine-Grained Zero-Shot Object Detection

TL;DR

This paper introduces the novel task of fine-grained zero-shot object detection, develops a specialized method called MSHC, and creates a new benchmark dataset FGZSD-Birds to evaluate performance in distinguishing highly similar classes.

Contribution

The paper defines FG-ZSD, proposes the MSHC method with multi-level semantics-aware embedding, and constructs the first FG-ZSD dataset FGZSD-Birds for evaluation.

Findings

MSHC outperforms existing ZSD models on FGZSD-Birds.

FGZSD-Birds contains 148,820 images across 1432 species.

The method effectively distinguishes fine-grained classes with minimal differences.

Abstract

Zero-shot object detection (ZSD) aims to leverage semantic descriptions to localize and recognize objects of both seen and unseen classes. Existing ZSD works are mainly coarse-grained object detection, where the classes are visually quite different, thus are relatively easy to distinguish. However, in real life we often have to face fine-grained object detection scenarios, where the classes are too similar to be easily distinguished. For example, detecting different kinds of birds, fishes, and flowers. In this paper, we propose and solve a new problem called Fine-Grained Zero-Shot Object Detection (FG-ZSD for short), which aims to detect objects of different classes with minute differences in details under the ZSD paradigm. We develop an effective method called MSHC for the FG-ZSD task, which is based on an improved two-stage detector and employs a multi-level semantics-aware embedding alignment loss, ensuring tight coupling between the visual and semantic spaces. Considering that existing ZSD datasets are not suitable for the new FG-ZSD task, we build the first FG-ZSD benchmark dataset FGZSD-Birds, which contains 148,820 images falling into 36 orders, 140 families, 579 genera and 1432 species. Extensive experiments on FGZSD-Birds show that our method outperforms existing ZSD models.