Taking Modality-free Human Identification as Zero-shot Learning

TL;DR

This paper introduces a novel zero-shot learning framework for modality-free human identification, capable of matching textual descriptions or attribute sets with visual identities across different modalities, addressing limitations of traditional image-to-image methods.

Contribution

It formulates the first modality-free human identification task as a scalable zero-shot learning problem, bridging visual and semantic modalities with a discriminative prototype learning approach.

Findings

Outperforms state-of-the-art methods on face and re-identification tasks.

Effectively bridges visual and semantic modalities.

Demonstrates robustness in modality-free identification scenarios.

Abstract

Human identification is an important topic in event detection, person tracking, and public security. There have been numerous methods proposed for human identification, such as face identification, person re-identification, and gait identification. Typically, existing methods predominantly classify a queried image to a specific identity in an image gallery set (I2I). This is seriously limited for the scenario where only a textual description of the query or an attribute gallery set is available in a wide range of video surveillance applications (A2I or I2A). However, very few efforts have been devoted towards modality-free identification, i.e., identifying a query in a gallery set in a scalable way. In this work, we take an initial attempt, and formulate such a novel Modality-Free Human Identification (named MFHI) task as a generic zero-shot learning model in a scalable way. Meanwhile, it is capable of bridging the visual and semantic modalities by learning a discriminative prototype of each identity. In addition, the semantics-guided spatial attention is enforced on visual modality to obtain representations with both high global category-level and local attribute-level discrimination. Finally, we design and conduct an extensive group of experiments on two common challenging identification tasks, including face identification and person re-identification, demonstrating that our method outperforms a wide variety of state-of-the-art methods on modality-free human identification.