Weakly Supervised Face Naming with Symmetry-Enhanced Contrastive Loss

TL;DR

This paper introduces SECLA and SECLA-B, novel weakly supervised models for face-name alignment that leverage symmetry-enhanced contrastive learning and a two-stage easy-to-hard training process, achieving state-of-the-art results.

Contribution

The paper proposes SECLA and SECLA-B, innovative contrastive learning models with a two-stage training framework for improved weakly supervised face-name alignment.

Findings

Achieved state-of-the-art results on Labeled Faces in the Wild and Celebrity Together datasets.

Demonstrated effectiveness of symmetry-enhanced contrastive loss in weakly supervised settings.

Showed that learning from easy to hard cases improves alignment performance.

Abstract

We revisit the weakly supervised cross-modal face-name alignment task; that is, given an image and a caption, we label the faces in the image with the names occurring in the caption. Whereas past approaches have learned the latent alignment between names and faces by uncertainty reasoning over a set of images and their respective captions, in this paper, we rely on appropriate loss functions to learn the alignments in a neural network setting and propose SECLA and SECLA-B. SECLA is a Symmetry-Enhanced Contrastive Learning-based Alignment model that can effectively maximize the similarity scores between corresponding faces and names in a weakly supervised fashion. A variation of the model, SECLA-B, learns to align names and faces as humans do, that is, learning from easy to hard cases to further increase the performance of SECLA. More specifically, SECLA-B applies a two-stage learning framework: (1) Training the model on an easy subset with a few names and faces in each image-caption pair. (2) Leveraging the known pairs of names and faces from the easy cases using a bootstrapping strategy with additional loss to prevent forgetting and learning new alignments at the same time. We achieve state-of-the-art results for both the augmented Labeled Faces in the Wild dataset and the Celebrity Together dataset. In addition, we believe that our methods can be adapted to other multimodal news understanding tasks.