Beyond Walking: A Large-Scale Image-Text Benchmark for Text-based Person Anomaly Search

TL;DR

This paper introduces a large-scale benchmark and a novel framework for text-based person anomaly search, enabling the identification of abnormal behaviors in real-world scenarios using natural language descriptions.

Contribution

It presents a new task, the Pedestrian Anomaly Behavior benchmark, and a cross-modal pose-aware framework for improved abnormal behavior retrieval.

Findings

Synthetic data enhances behavior retrieval accuracy.

The pose-aware method achieves 84.93% recall@1.

The benchmark covers diverse normal and abnormal actions.

Abstract

Text-based person search aims to retrieve specific individuals across camera networks using natural language descriptions. However, current benchmarks often exhibit biases towards common actions like walking or standing, neglecting the critical need for identifying abnormal behaviors in real-world scenarios. To meet such demands, we propose a new task, text-based person anomaly search, locating pedestrians engaged in both routine or anomalous activities via text. To enable the training and evaluation of this new task, we construct a large-scale image-text Pedestrian Anomaly Behavior (PAB) benchmark, featuring a broad spectrum of actions, e.g., running, performing, playing soccer, and the corresponding anomalies, e.g., lying, being hit, and falling of the same identity. The training set of PAB comprises 1,013,605 synthesized image-text pairs of both normalities and anomalies, while the test set includes 1,978 real-world image-text pairs. To validate the potential of PAB, we introduce a cross-modal pose-aware framework, which integrates human pose patterns with identity-based hard negative pair sampling. Extensive experiments on the proposed benchmark show that synthetic training data facilitates the fine-grained behavior retrieval, and the proposed pose-aware method arrives at 84.93% recall@1 accuracy, surpassing other competitive methods. The dataset, model, and code are available at https://github.com/Shuyu-XJTU/CMP.