ArtFace: Towards Historical Portrait Face Identification via Model Adaptation

TL;DR

This paper explores how fine-tuning foundation models and combining their embeddings with traditional facial recognition methods can significantly improve the identification of sitters in historical paintings, addressing challenges like style variation and domain shift.

Contribution

It introduces a novel approach that integrates foundation model embeddings with conventional facial recognition networks to enhance artwork face identification.

Findings

Foundation models improve recognition accuracy in paintings.

Combining embeddings outperforms traditional methods.

Significant gains over state-of-the-art techniques.

Abstract

Identifying sitters in historical paintings is a key task for art historians, offering insight into their lives and how they chose to be seen. However, the process is often subjective and limited by the lack of data and stylistic variations. Automated facial recognition is capable of handling challenging conditions and can assist, but while traditional facial recognition models perform well on photographs, they struggle with paintings due to domain shift and high intra-class variation. Artistic factors such as style, skill, intent, and influence from other works further complicate recognition. In this work, we investigate the potential of foundation models to improve facial recognition in artworks. By fine-tuning foundation models and integrating their embeddings with those from conventional facial recognition networks, we demonstrate notable improvements over current state-of-the-art methods. Our results show that foundation models can bridge the gap where traditional methods are ineffective. Paper page at https://www.idiap.ch/paper/artface/

Tables

Table 1. Table 1 : Performance of IResNet100 variants with and without tuning and fusion.

Model	EER$\downarrow$	TAR@0.1%FAR$\uparrow$	TAR@1%FAR$\uparrow$
IResNet100-Base	14.0%	29.9%	55.1%
IResNet100-Tuned	13.5%	36.5%	53.7%
IResNet100-Fusion (Base+Tuned)	13.5%	33.6%	58.4%

Table 2. Table 2 : Performance of loss functions for tuning CLIP. Models with Hard Negative Mining (HN) perform better overall.

Model	EER $\downarrow$	TAR@0.1%FAR$\uparrow$	TAR@1%FAR$\uparrow$
CLIP-Base	17.9%	8.4%	33.2%
CLIP-LoRA(Triplet, HN)	13.1%	17.8%	43.5%
CLIP-LoRA(Triplet, w/o HN)	13.9%	16.8%	43.9%

Table 3. Table 3 : Performance Comparison of Base, Tuned models, Fusion, and COTS FR Systems. Fusion enhances overall accuracy.

Model	EER$\downarrow$	TAR@0.1%FAR$\uparrow$	TAR@1%FAR$\uparrow$
COTS FR system	12.6%	34.3%	58.1%
CLIP-Base	17.9%	8.4%	33.2%
IResNet100-Base	14.0%	29.9%	55.1%
CLIP-Base + IResNet100-Base	13.1%	29.0%	54.7%
CLIP-Base + IResNet100-Tuned	12.6%	35.1%	57.9%
CLIP-LoRA + IResNet100-Base	11.1%	34.6%	62.6%
CLIP-LoRA + IResNet100-Tuned	10.7%	39.7%	62.15%
CLIP-LoRA + IResNet100-Base + IResNet100-Tuned	9.9%	39.7%	65.9%

Full text

ArtFace: Towards Historical Portrait Face Identification via Model Adaptation

Francois Poh, Anjith George and Sébastien Marcel

Idiap Research Institute

Rue Marconi 19, CH - 1920, Martigny, Switzerland

francois.poh22@imperial.ac.uk, {anjith.george, marcel}@idiap.ch

1 Introduction

Identifying sitters in historical paintings is a key task for art historians, offering insight into their lives and how they chose to be seen [11]. However, the process is often subjective and limited by the lack of data and stylistic variations. Automated facial recognition is capable of handling challenging conditions and can assist, but while traditional facial recognition models perform well on photographs [7], they struggle with paintings due to domain shift and high intra-class variation [5]. Artistic factors such as style, skill, intent [8, 11], and influence from other works [1] further complicate recognition.

The early work of Srinivasan et al. [14] approached the problem using local and anthropometric facial features to statistically assess similarity between portrait pairs. More recent approaches rely heavily on deep learning, particularly convolutional neural networks (CNNs). Gupta et al. [5] used Siamese CNNs to compare controversial historical portraits, leveraging style transfer techniques to augment training data while preserving essential facial content. Building on this, Huber et al.  [8] fine-tuned a ResNet-100 model on an augmented dataset and introduced a confidence measure by estimating uncertainty in face comparison scores. Salavessa et al.  [11] combined deep learning with handcrafted facial feature ratios, using a VGG-based approach.

In this work, we investigate the potential of foundation models to improve facial recognition in artworks. By fine-tuning foundation models and integrating their embeddings with those from conventional facial recognition networks, we demonstrate notable improvements over current state-of-the-art methods. Our results show that foundation models can bridge the gap where traditional methods are ineffective. Paper page at 111https://www.idiap.ch/paper/artface/.

Contributions

We address the task of sitter identification through the following contributions: (i) Fine‑tuning the vision–language foundation model CLIP [10] with LoRA [6] on portrait face images; (ii) Complementing it with embeddings from an adapted face recognition network (AntelopeV2/IResNet100 [2]); and (iii) Fusing the embeddings via a simple yet effective normalised concatenation strategy. Comprehensive experiments on the public Historical Faces dataset [8] show that our method reduces baseline error rates and outperforms prior approaches.

2 Proposed Method

CLIP Foundation Model: CLIP is a vision–language foundation model trained on 400 million image–text pairs collected from the internet [10]. It learns to align images and text in a shared embedding space, enabling zero-shot transfer to a wide range of tasks such as image classification and visual understanding. Since foundation models are trained on diverse data, they capture broad concepts, detect stylistic differences, and can use contextual information relevant to paintings that traditional facial recognition overlooks. Prior work has shown that combining CLIP embeddings with those from facial recognition models yields modest improvements on photographic data [13], and this benefit can extend to painted portraits as well. For our experiments, we use the CLIP ViT-B/16 model from OpenAI, selected for its balance between tunability and computational efficiency.

Face Recognition Model: AntelopeV2 is a state-of-the-art face recognition (FR) model built on the IResNet100 architecture [2] and trained on the large-scale Glint360k dataset. It achieves high accuracy and robustness across challenging variations in pose, age, and lighting, making it a strong and widely adopted baseline in face verification benchmarks. In addition to these models, we also use a Commercial off-the-shelf face recognition model as another baseline (COTS).

Model Adaptation: Both the CLIP and face recognition (FR) models are adapted using portrait images to better align with the domain of historical paintings, as shown in Fig. 1. For CLIP, we apply Low-Rank Adaptation (LoRA) [6], inserting LoRA layers into the query (Q) and value (V) matrices of the attention mechanism. This approach introduces low-rank updates, allowing efficient fine-tuning with significantly fewer trainable parameters and reduced memory usage, while maintaining performance. For the IResNet100 model, adaptation is minimal; only the final linear layer is fine-tuned to adjust the embeddings for the new domain.

Fusion: Embeddings from the tuned and untuned IResNet100 models, along with the tuned CLIP model, are first individually normalised, then concatenated and re-normalised. This process effectively performs feature fusion, combining complementary information from each model.

3 Experiments

Dataset: The Historical Faces dataset [8] consists of 766 paintings of 210 different sitters. The paintings are preprocessed by detecting facial key points and aligning and cropping to 112x112 for both IResNet100 and CLIP. This was split for training and evaluation by identity in a 60:20:20 ratio for the train, validation, and test set, respectively.

Training Details: The CLIP-LoRA model is trained using triplet loss [12] with cosine distance and a margin of 0.5, with a batch size of 48. Training employs early stopping with a patience of 10 epochs to prevent overfitting. Hard negative mining is performed at each epoch: 30% of the negative samples are selected from the top 50 hardest negatives per anchor, while the remaining 70% are randomly drawn from the next 450 hardest examples. The model is optimised using Adam with a learning rate of 1e-5. LoRA layers, with rank 16, are applied to the query (Q) and value (V) matrices in all transformer layers, implemented using the PEFT library [9]. IResNet100 uses the same method and parameters for training, but no LoRA layers are applied; instead, only tuning the final linear layer. The final output score is the cosine similarity between the concatenated embeddings.

3.1 Experimental Results

Results with IResNet100 Tuning: Table 1 demonstrates that tuning IResNet100 primarily enhances TAR222True Accept Rate (TAR) = 1 - False Non-Match Rate (FNMR) at low FAR333False Accept Rate (FAR) = False Match Rate (FMR) values ($<$1%), with only modest improvements in EER444Equal Error Rate (EER) = Error Rate when FMR=FNMR. While fusion performs slightly worse than the tuned model in low-FAR TAR, it consistently outperforms both the base and tuned models overall.

Results with CLIP-LoRA Tuning: In this section, we describe the fine-tuning of CLIP-LoRA using triplet loss, both with and without hard negative mining. All other relevant parameters remain unchanged from previous experiments. Although the hard negative mining variant does not achieve higher TAR at all FAR levels, it shows improved performance above approximately 1% FAR. As shown in Table 2, this approach also leads to better EER.

Results with Fusion: Table 3 compares the performance of fusion methods with individual IResNet100 and CLIP models on the Historical Faces test split. Although the base CLIP model underperforms compared to IResNet100, fusion leads to improvements in EER. Fine-tuning CLIP-LoRA further enhances performance across all four metrics, with additional gains observed when the tuned IResNet100 is included in the fusion. The corresponding ROC curves are presented in Fig. 3.

Discussions: Tuning IResNet100 on the Historical Faces dataset yields only modest gains, but fusing tuned and untuned versions outperforms either alone, suggesting combining representations can help, even within a single architecture.

CLIP-LoRA tuning shows triplet loss with hard negative mining performs best, significantly improving over base CLIP. Other loss functions underperform, likely due to limited data, especially in terms of images per identity.

Fusion experiments show that combining foundation models with conventional face recognition networks improves performance across all evaluation metrics, especially in low false acceptance rate (FAR) scenarios. This indicates that foundation models capture valuable information from portraits that conventional networks may overlook.

Visualisations: To better understand both successful and failed matches, we show sample images along with their similarity scores in Fig. 2. Genuine comparisons can fail in challenging cases, particularly when there are significant differences in artistic style or medium, sitter age, and often compounded when multiple factors overlap. Conversely, some impostor comparisons yield surprisingly high similarity scores. This typically occurs when artists base their work on earlier portraits or when sitters resemble each other and are depicted in a similar style or by the same artist.

4 Conclusions

In this work, we show that lightweight tuning of vision–language foundation models [6], combined with domain-adapted face recognition networks, can effectively bridge the domain gap between photographs and paintings. Our fusion approach achieves state-of-the-art accuracy in sitter identification. Face recognition on artworks remains a particularly difficult task compared to traditional FR due to the scarcity of labelled data, stylistic variation, and the interpretive nature of portraiture. However, the results show that adapting modern architectures to this setting is feasible and promising. This opens up new research avenues, including synthetic data generation [4] to augment the limited training set and heterogeneous domain adaptation techniques [3] to improve generalisation across visual domains.

5 Acknowledgements

This research was supported by the InterArt project, funded by Loterie Romande.