SPOT-Face: Forensic Face Identification using Attention Guided Optimal Transport

TL;DR

This paper introduces SPOT-Face, a novel graph-based framework utilizing attention-guided optimal transport for cross-domain forensic face identification, effectively matching skeletons and sketches to faces with improved accuracy.

Contribution

The paper presents a new superpixel graph-based method with attention-guided optimal transport for cross-domain forensic face identification, addressing limitations of existing deep learning approaches.

Findings

Significant improvement in identification metrics over baselines

Effective matching of skulls and sketches to faces

Validated on two public datasets with strong results

Abstract

Person identification in forensic investigations becomes very challenging when common identification means for DNA (i.e., hair strands, soft tissue) are not available. Current methods utilize deep learning methods for face recognition. However, these methods lack effective mechanisms to model cross-domain structural correspondence between two different forensic modalities. In this paper, we introduce a SPOT-Face, a superpixel graph-based framework designed for cross-domain forensic face identification of victims using their skeleton and sketch images. Our unified framework involves constructing a superpixel-based graph from an image and then using different graph neural networks(GNNs) backbones to extract the embeddings of these graphs, while cross-domain correspondence is established through attention-guided optimal transport mechanism. We have evaluated our proposed framework on two publicly available dataset: IIT\_Mandi\_S2F (S2F) and CUFS. Extensive experiments were conducted to evaluate our proposed framework. The experimental results show significant improvement in identification metrics ( i.e., Recall, mAP) over existing graph-based baselines. Furthermore, our framework demonstrates to be highly effective for matching skulls and sketches to faces in forensic investigations.