One-shot lip-based biometric authentication: extending behavioral features with authentication phrase information

TL;DR

This paper enhances lip-based biometric authentication by incorporating speech content recognition to prevent replay attacks, using a customized dataset and a siamese neural network with triplet loss, achieving low error rates.

Contribution

It introduces a novel one-shot LBBA method that models speech content alongside behavioral features, improving security against replay attacks.

Findings

Achieved 3.2% FAR and 3.8% FRR on customized dataset.

Demonstrated the effectiveness of speech content modeling in LBBA.

Quantified the importance of behavioral versus physical features.

Abstract

Lip-based biometric authentication (LBBA) is an authentication method based on a person's lip movements during speech in the form of video data captured by a camera sensor. LBBA can utilize both physical and behavioral characteristics of lip movements without requiring any additional sensory equipment apart from an RGB camera. State-of-the-art (SOTA) approaches use one-shot learning to train deep siamese neural networks which produce an embedding vector out of these features. Embeddings are further used to compute the similarity between an enrolled user and a user being authenticated. A flaw of these approaches is that they model behavioral features as style-of-speech without relation to what is being said. This makes the system vulnerable to video replay attacks of the client speaking any phrase. To solve this problem we propose a one-shot approach which models behavioral features to discriminate against what is being said in addition to style-of-speech. We achieve this by customizing the GRID dataset to obtain required triplets and training a siamese neural network based on 3D convolutions and recurrent neural network layers. A custom triplet loss for batch-wise hard-negative mining is proposed. Obtained results using an open-set protocol are 3.2% FAR and 3.8% FRR on the test set of the customized GRID dataset. Additional analysis of the results was done to quantify the influence and discriminatory power of behavioral and physical features for LBBA.