Text-Guided Face Recognition using Multi-Granularity Cross-Modal Contrastive Learning

TL;DR

This paper introduces a novel text-guided face recognition framework that leverages natural language facial descriptions and multi-granularity cross-modal contrastive learning to improve recognition accuracy, especially in low-quality surveillance images.

Contribution

It proposes a face-caption alignment module with contrastive losses and a face-caption fusion module for enhanced multimodal feature learning, addressing semantic gaps and textual ambiguities.

Findings

Significant performance improvements on low-quality images.

Outperforms existing face recognition models and benchmarks.

Effective integration of facial attributes via natural language enhances recognition.

Abstract

State-of-the-art face recognition (FR) models often experience a significant performance drop when dealing with facial images in surveillance scenarios where images are in low quality and often corrupted with noise. Leveraging facial characteristics, such as freckles, scars, gender, and ethnicity, becomes highly beneficial in improving FR performance in such scenarios. In this paper, we introduce text-guided face recognition (TGFR) to analyze the impact of integrating facial attributes in the form of natural language descriptions. We hypothesize that adding semantic information into the loop can significantly improve the image understanding capability of an FR algorithm compared to other soft biometrics. However, learning a discriminative joint embedding within the multimodal space poses a considerable challenge due to the semantic gap in the unaligned image-text representations, along with the complexities arising from ambiguous and incoherent textual descriptions of the face. To address these challenges, we introduce a face-caption alignment module (FCAM), which incorporates cross-modal contrastive losses across multiple granularities to maximize the mutual information between local and global features of the face-caption pair. Within FCAM, we refine both facial and textual features for learning aligned and discriminative features. We also design a face-caption fusion module (FCFM) that applies fine-grained interactions and coarse-grained associations among cross-modal features. Through extensive experiments conducted on three face-caption datasets, proposed TGFR demonstrates remarkable improvements, particularly on low-quality images, over existing FR models and outperforms other related methods and benchmarks.