Seamless Multimodal Biometrics for Continuous Personalised Wellbeing Monitoring

TL;DR

This paper advances in-vehicle biometric and wellbeing monitoring by developing deep learning solutions for ECG and face biometrics, multimodal emotion recognition, and secure, self-supervised data processing for autonomous vehicle applications.

Contribution

It introduces novel end-to-end deep learning methods for ECG and face biometrics, enhances multimodal emotion and activity recognition, and proposes integrated template security and self-supervised learning techniques.

Findings

Improved ECG biometric identification and verification performance.

Enhanced face recognition robustness to occlusions and masks.

Effective multimodal emotion and activity recognition in vehicle scenarios.

Abstract

Artificially intelligent perception is increasingly present in the lives of every one of us. Vehicles are no exception, (...) In the near future, pattern recognition will have an even stronger role in vehicles, as self-driving cars will require automated ways to understand what is happening around (and within) them and act accordingly. (...) This doctoral work focused on advancing in-vehicle sensing through the research of novel computer vision and pattern recognition methodologies for both biometrics and wellbeing monitoring. The main focus has been on electrocardiogram (ECG) biometrics, a trait well-known for its potential for seamless driver monitoring. Major efforts were devoted to achieving improved performance in identification and identity verification in off-the-person scenarios, well-known for increased noise and variability. Here, end-to-end deep learning ECG biometric solutions were proposed and important topics were addressed such as cross-database and long-term performance, waveform relevance through explainability, and interlead conversion. Face biometrics, a natural complement to the ECG in seamless unconstrained scenarios, was also studied in this work. The open challenges of masked face recognition and interpretability in biometrics were tackled in an effort to evolve towards algorithms that are more transparent, trustworthy, and robust to significant occlusions. Within the topic of wellbeing monitoring, improved solutions to multimodal emotion recognition in groups of people and activity/violence recognition in in-vehicle scenarios were proposed. At last, we also proposed a novel way to learn template security within end-to-end models, dismissing additional separate encryption processes, and a self-supervised learning approach tailored to sequential data, in order to ensure data security and optimal performance. (...)