Detect Only What You Specify : Object Detection with Linguistic Target

TL;DR

This paper introduces a language-guided object detection approach that detects only specified objects based on natural language input, enabling more targeted and context-aware detection in images.

Contribution

The paper proposes the Language-Targeted Detector (LTD), a Transformer-based model that incorporates linguistic context for selective object detection, addressing limitations of traditional detectors.

Findings

LTD improves detection accuracy when guided by textual input.

The model effectively grounds language to visual objects in the COCO dataset.

LTD demonstrates the ability to detect only specified objects based on language cues.

Abstract

Object detection is a computer vision task of predicting a set of bounding boxes and category labels for each object of interest in a given image. The category is related to a linguistic symbol such as 'dog' or 'person' and there should be relationships among them. However the object detector only learns to classify the categories and does not treat them as the linguistic symbols. Multi-modal models often use the pre-trained object detector to extract object features from the image, but the models are separated from the detector and the extracted visual features does not change with their linguistic input. We rethink the object detection as a vision-and-language reasoning task. We then propose targeted detection task, where detection targets are given by a natural language and the goal of the task is to detect only all the target objects in a given image. There are no detection if the target is not given. Commonly used modern object detectors have many hand-designed components like anchor and it is difficult to fuse the textual inputs into the complex pipeline. We thus propose Language-Targeted Detector (LTD) for the targeted detection based on a recently proposed Transformer-based detector. LTD is a encoder-decoder architecture and our conditional decoder allows the model to reason about the encoded image with the textual input as the linguistic context. We evaluate detection performances of LTD on COCO object detection dataset and also show that our model improves the detection results with the textual input grounding to the visual object.