Head Similarity: Modeling Structured Whole-Head Appearance Beyond Face Recognition

TL;DR

This paper introduces Head Similarity, a novel approach for modeling structured whole-head appearance to enable identity recognition beyond facial cues, especially under challenging viewing conditions.

Contribution

It extends traditional face recognition by explicitly capturing intra-identity appearance variations and hierarchical similarity, enabling more robust identity comparisons.

Findings

Conventional face models fail to capture appearance-dependent similarity.

The proposed framework effectively models structured whole-head similarity.

Experiments demonstrate improved recognition under occlusion and non-frontal views.

Abstract

Many vision applications require identity consistency beyond strict biometric recognition, especially under non-frontal views or when facial cues are missing. However, conventional face recognition models enforce intra-identity invariance, collapsing appearance variations such as hairstyle or styling changes into a single representation, limiting their use in appearance-sensitive scenarios. To address this limitation, we introduce Head Similarity, a new formulation that extends identity-centric recognition to structured whole-head similarity modeling. Our approach explicitly captures intra-identity appearance variation and enforces hierarchical similarity ordering across identity and appearance states, enabling meaningful comparison even under occlusion or rear-view conditions. We construct a large-scale benchmark from long-form videos with weakly-supervised appearance states, covering diverse poses, occlusions, and temporal changes. As a first step, we develop a simple yet effective framework that jointly models identity discrimination and appearance-sensitive similarity through hierarchical supervision and identity-aware distillation. Experiments show that conventional face recognition models fail to capture appearance-dependent similarity, while our approach demonstrates the feasibility of structured whole-head similarity modeling.