Person Re-Identification with Vision and Language

TL;DR

This paper introduces a novel person re-identification approach combining vision and natural language, enhancing matching accuracy by leveraging joint models and new language annotations on standard datasets.

Contribution

It presents a joint vision-language model based on CCA and CNN architectures, along with new natural language annotations for CUHK03 and VIPeR datasets, improving Re-ID performance.

Findings

Natural language descriptions outperform attributes in Re-ID.

Joint vision and language models significantly improve accuracy.

CNN-based methods outperform LSTM in this context.

Abstract

In this paper we propose a new approach to person re-identification using images and natural language descriptions. We propose a joint vision and language model based on CCA and CNN architectures to match across the two modalities as well as to enrich visual examples for which there are no language descriptions. We also introduce new annotations in the form of natural language descriptions for two standard Re-ID benchmarks, namely CUHK03 and VIPeR. We perform experiments on these two datasets with techniques based on CNN, hand-crafted features as well as LSTM for analysing visual and natural description data. We investigate and demonstrate the advantages of using natural language descriptions compared to attributes as well as CNN compared to LSTM in the context of Re-ID. We show that the joint use of language and vision can significantly improve the state-of-the-art performance on standard Re-ID benchmarks.