Attention mechanisms for physiological signal deep learning: which attention should we take?

TL;DR

This study compares four attention mechanisms across different CNN architectures for physiological signal prediction tasks, finding that attention improves performance and convergence, with specific mechanisms excelling in classification and regression.

Contribution

It provides an experimental analysis of multiple attention mechanisms and CNN architectures, revealing their complementary effects and optimal configurations for physiological signal deep learning.

Findings

Spatial attention improves classification performance.

Channel attention reduces regression error.

Attention mechanisms enhance convergence speed.

Abstract

Attention mechanisms are widely used to dramatically improve deep learning model performance in various fields. However, their general ability to improve the performance of physiological signal deep learning model is immature. In this study, we experimentally analyze four attention mechanisms (e.g., squeeze-and-excitation, non-local, convolutional block attention module, and multi-head self-attention) and three convolutional neural network (CNN) architectures (e.g., VGG, ResNet, and Inception) for two representative physiological signal prediction tasks: the classification for predicting hypotension and the regression for predicting cardiac output (CO). We evaluated multiple combinations for performance and convergence of physiological signal deep learning model. Accordingly, the CNN models with the spatial attention mechanism showed the best performance in the classification problem, whereas the channel attention mechanism achieved the lowest error in the regression problem. Moreover, the performance and convergence of the CNN models with attention mechanisms were better than stand-alone self-attention models in both problems. Hence, we verified that convolutional operation and attention mechanisms are complementary and provide faster convergence time, despite the stand-alone self-attention models requiring fewer parameters.