Video-based Person Re-identification with Spatial and Temporal Memory Networks

TL;DR

This paper introduces Spatial and Temporal Memory Networks (STMN) for video-based person re-identification, effectively handling distractors by leveraging learned spatial and temporal patterns to improve sequence-level person representations.

Contribution

The novel STMN architecture models spatial and temporal distractors with dedicated memories, enhancing feature refinement and aggregation for better re-identification accuracy.

Findings

Outperforms existing methods on MARS, DukeMTMC-VideoReID, and LS-VID benchmarks.

Effectively handles spatial distractors like background clutter.

Successfully models temporal patterns such as partial occlusions.

Abstract

Video-based person re-identification (reID) aims to retrieve person videos with the same identity as a query person across multiple cameras. Spatial and temporal distractors in person videos, such as background clutter and partial occlusions over frames, respectively, make this task much more challenging than image-based person reID. We observe that spatial distractors appear consistently in a particular location, and temporal distractors show several patterns, e.g., partial occlusions occur in the first few frames, where such patterns provide informative cues for predicting which frames to focus on (i.e., temporal attentions). Based on this, we introduce a novel Spatial and Temporal Memory Networks (STMN). The spatial memory stores features for spatial distractors that frequently emerge across video frames, while the temporal memory saves attentions which are optimized for typical temporal patterns in person videos. We leverage the spatial and temporal memories to refine frame-level person representations and to aggregate the refined frame-level features into a sequence-level person representation, respectively, effectively handling spatial and temporal distractors in person videos. We also introduce a memory spread loss preventing our model from addressing particular items only in the memories. Experimental results on standard benchmarks, including MARS, DukeMTMC-VideoReID, and LS-VID, demonstrate the effectiveness of our method.