Discriminative Learning of Open-Vocabulary Object Retrieval and Localization by Negative Phrase Augmentation

TL;DR

This paper introduces Query-Adaptive R-CNN and negative phrase augmentation to enable open-vocabulary object retrieval and localization from large image collections, significantly expanding the capabilities of traditional detectors.

Contribution

It presents a novel extension of Faster R-CNN for open-vocabulary queries and a negative phrase augmentation technique for discriminative training.

Findings

Retrieves and localizes objects from one million images in 0.5 seconds

Achieves high precision in open-vocabulary object localization

Effectively mines hard negative samples for improved training

Abstract

Thanks to the success of object detection technology, we can retrieve objects of the specified classes even from huge image collections. However, the current state-of-the-art object detectors (such as Faster R-CNN) can only handle pre-specified classes. In addition, large amounts of positive and negative visual samples are required for training. In this paper, we address the problem of open-vocabulary object retrieval and localization, where the target object is specified by a textual query (e.g., a word or phrase). We first propose Query-Adaptive R-CNN, a simple extension of Faster R-CNN adapted to open-vocabulary queries, by transforming the text embedding vector into an object classifier and localization regressor. Then, for discriminative training, we then propose negative phrase augmentation (NPA) to mine hard negative samples which are visually similar to the query and at the same time semantically mutually exclusive of the query. The proposed method can retrieve and localize objects specified by a textual query from one million images in only 0.5 seconds with high precision.