LVFace: Progressive Cluster Optimization for Large Vision Models in Face Recognition

TL;DR

LVFace introduces a ViT-based face recognition model with Progressive Cluster Optimization, achieving state-of-the-art results and demonstrating scalability and robustness in large-scale and real-world scenarios.

Contribution

The paper presents LVFace, a novel ViT-based face recognition framework with PCO, improving performance and stability over CNN-inspired training methods.

Findings

LVFace surpasses leading face recognition models on multiple benchmarks.

LVFace achieves first place in the ICCV 2021 Masked Face Recognition Challenge.

LVFace demonstrates scalability and compatibility with mainstream vision and language models.

Abstract

Vision Transformers (ViTs) have revolutionized large-scale visual modeling, yet remain underexplored in face recognition (FR) where CNNs still dominate. We identify a critical bottleneck: CNN-inspired training paradigms fail to unlock ViT's potential, leading to suboptimal performance and convergence instability.To address this challenge, we propose LVFace, a ViT-based FR model that integrates Progressive Cluster Optimization (PCO) to achieve superior results. Specifically, PCO sequentially applies negative class sub-sampling (NCS) for robust and fast feature alignment from random initialization, feature expectation penalties for centroid stabilization, performing cluster boundary refinement through full-batch training without NCS constraints. LVFace establishes a new state-of-the-art face recognition baseline, surpassing leading approaches such as UniFace and TopoFR across multiple benchmarks. Extensive experiments demonstrate that LVFace delivers consistent performance gains, while exhibiting scalability to large-scale datasets and compatibility with mainstream VLMs and LLMs. Notably, LVFace secured 1st place in the ICCV 2021 Masked Face Recognition (MFR)-Ongoing Challenge (March 2025), proving its efficacy in real-world scenarios. Project is available at https://github.com/bytedance/LVFace.