FaceCoresetNet: Differentiable Coresets for Face Set Recognition

TL;DR

This paper introduces FaceCoresetNet, a novel differentiable coreset selection method for face set recognition that balances quality and diversity, achieving state-of-the-art results on standard benchmarks.

Contribution

It proposes a differentiable coreset selection framework using Gumbel-Softmax for face recognition, enabling end-to-end training and improved set representation.

Findings

Achieves new state-of-the-art on IJB-B and IJB-C datasets.

Order-invariant and linear complexity in set size.

Effectively balances quality and diversity in face set aggregation.

Abstract

In set-based face recognition, we aim to compute the most discriminative descriptor from an unbounded set of images and videos showing a single person. A discriminative descriptor balances two policies when aggregating information from a given set. The first is a quality-based policy: emphasizing high-quality and down-weighting low-quality images. The second is a diversity-based policy: emphasizing unique images in the set and down-weighting multiple occurrences of similar images as found in video clips which can overwhelm the set representation. This work frames face-set representation as a differentiable coreset selection problem. Our model learns how to select a small coreset of the input set that balances quality and diversity policies using a learned metric parameterized by the face quality, optimized end-to-end. The selection process is a differentiable farthest-point sampling (FPS) realized by approximating the non-differentiable Argmax operation with differentiable sampling from the Gumbel-Softmax distribution of distances. The small coreset is later used as queries in a self and cross-attention architecture to enrich the descriptor with information from the whole set. Our model is order-invariant and linear in the input set size. We set a new SOTA to set face verification on the IJB-B and IJB-C datasets. Our code is publicly available.