CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models

TL;DR

This paper introduces CLIP4Sketch, a method that uses diffusion models to generate diverse, high-quality sketches from mugshots, significantly improving face recognition accuracy in forensic sketch matching tasks.

Contribution

We propose a novel diffusion model-based approach to generate synthetic sketches, enhancing dataset diversity and face recognition performance in sketch-to-mugshot matching.

Findings

Synthetic sketches improve recognition accuracy.

Diffusion-generated data outperforms GAN-based datasets.

Enhanced dataset leads to better forensic face matching.

Abstract

Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.