Fusion of Physiological and Behavioural Signals on SPD Manifolds with Application to Stress and Pain Detection

TL;DR

This paper introduces a geometric framework using SPD matrices to capture cross-modality correlations in physiological and behavioural signals for stress and pain detection, employing tangent space mapping and LSTM networks.

Contribution

It presents a novel SPD manifold-based approach that models cross-modality correlations and applies tangent space mapping with LSTM for improved classification.

Findings

Achieved state-of-the-art results on stress detection datasets.

Effectively models cross-modality correlations with SPD matrices.

Demonstrates the effectiveness of tangent space mapping for classification.

Abstract

Existing multimodal stress/pain recognition approaches generally extract features from different modalities independently and thus ignore cross-modality correlations. This paper proposes a novel geometric framework for multimodal stress/pain detection utilizing Symmetric Positive Definite (SPD) matrices as a representation that incorporates the correlation relationship of physiological and behavioural signals from covariance and cross-covariance. Considering the non-linearity of the Riemannian manifold of SPD matrices, well-known machine learning techniques are not suited to classify these matrices. Therefore, a tangent space mapping method is adopted to map the derived SPD matrix sequences to the vector sequences in the tangent space where the LSTM-based network can be applied for classification. The proposed framework has been evaluated on two public multimodal datasets, achieving both the state-of-the-art results for stress and pain detection tasks.