Efficient-Husformer: Efficient Multimodal Transformer Hyperparameter Optimization for Stress and Cognitive Loads

TL;DR

Efficient-Husformer introduces a hyperparameter-optimized Transformer architecture for multimodal stress detection, achieving significant accuracy improvements with a compact model on physiological datasets.

Contribution

This work presents a novel hyperparameter optimization framework for Transformer models, improving stress detection accuracy and model efficiency in multimodal physiological signal analysis.

Findings

Achieved 88.41% and 92.61% accuracy on WESAD and CogLoad datasets.

Designed a structured search space for hyperparameter tuning.

Developed a compact model with approximately 30,000 parameters.

Abstract

Transformer-based models have gained considerable attention in the field of physiological signal analysis. They leverage long-range dependencies and complex patterns in temporal signals, allowing them to achieve performance superior to traditional RNN and CNN models. However, they require high computational intensity and memory demands. In this work, we present Efficient-Husformer, a novel Transformer-based architecture developed with hyperparameter optimization (HPO) for multi-class stress detection across two multimodal physiological datasets (WESAD and CogLoad). The main contributions of this work are: (1) the design of a structured search space, targeting effective hyperparameter optimization; (2) a comprehensive ablation study evaluating the impact of architectural decisions; (3) consistent performance improvements over the original Husformer, with the best configuration achieving an accuracy of 88.41 and 92.61 (improvements of 13.83% and 6.98%) on WESAD and CogLoad datasets, respectively. The best-performing configuration is achieved with the (L + dm) or (L + FFN) modality combinations, using a single layer, 3 attention heads, a model dimension of 18/30, and FFN dimension of 120/30, resulting in a compact model with only about 30k parameters.