TVPR: Text-to-Video Person Retrieval and a New Benchmark

TL;DR

This paper introduces the TVPR task and a new dataset for text-to-video person retrieval, along with a novel learning strategy that significantly improves retrieval performance by leveraging cross-modal representations.

Contribution

It presents the first video-based text person retrieval method and constructs a large-scale dataset with natural language annotations for this task.

Findings

Achieved state-of-the-art results on the TVPReid dataset.

Demonstrated the effectiveness of the MFGF strategy in cross-modal learning.

Provided a new benchmark for future research in text-to-video person retrieval.

Abstract

Most existing methods for text-based person retrieval focus on text-to-image person retrieval. Nevertheless, due to the lack of dynamic information provided by isolated frames, the performance is hampered when the person is obscured or variable motion details are missed in isolated frames. To overcome this, we propose a novel Text-to-Video Person Retrieval (TVPR) task. Since there is no dataset or benchmark that describes person videos with natural language, we construct a large-scale cross-modal person video dataset containing detailed natural language annotations, termed as Text-to-Video Person Re-identification (TVPReid) dataset. In this paper, we introduce a Multielement Feature Guided Fragments Learning (MFGF) strategy, which leverages the cross-modal text-video representations to provide strong text-visual and text-motion matching information to tackle uncertain occlusion conflicting and variable motion details. Specifically, we establish two potential cross-modal spaces for text and video feature collaborative learning to progressively reduce the semantic difference between text and video. To evaluate the effectiveness of the proposed MFGF, extensive experiments have been conducted on TVPReid dataset. To the best of our knowledge, MFGF is the first successful attempt to use video for text-based person retrieval task and has achieved state-of-the-art performance on TVPReid dataset. The TVPReid dataset will be publicly available to benefit future research.