Person Search with Natural Language Description

TL;DR

This paper introduces a new approach for person search using natural language descriptions, including a large-scale dataset and a novel neural network model that outperforms existing methods.

Contribution

The paper presents the CUHK-PEDES dataset and a GNA-RNN model for natural language-based person search, advancing beyond image or attribute-based methods.

Findings

GNA-RNN achieves state-of-the-art performance on the new dataset.

The CUHK-PEDES dataset enables benchmarking of text-based person search.

Natural language descriptions improve search flexibility and accuracy.

Abstract

Searching persons in large-scale image databases with the query of natural language description has important applications in video surveillance. Existing methods mainly focused on searching persons with image-based or attribute-based queries, which have major limitations for a practical usage. In this paper, we study the problem of person search with natural language description. Given the textual description of a person, the algorithm of the person search is required to rank all the samples in the person database then retrieve the most relevant sample corresponding to the queried description. Since there is no person dataset or benchmark with textual description available, we collect a large-scale person description dataset with detailed natural language annotations and person samples from various sources, termed as CUHK Person Description Dataset (CUHK-PEDES). A wide range of possible models and baselines have been evaluated and compared on the person search benchmark. An Recurrent Neural Network with Gated Neural Attention mechanism (GNA-RNN) is proposed to establish the state-of-the art performance on person search.