IsoNet: Causal Analysis of Multimodal Transformers for Neuromuscular Gesture Classification

TL;DR

This paper evaluates multimodal fusion strategies in transformer architectures for neuromuscular gesture classification, demonstrating that attention-based hierarchical fusion significantly improves accuracy and provides insights into modality interactions.

Contribution

It introduces an Isolation Network to quantify modality interactions and compares fusion strategies across multiple architectures, advancing understanding of multimodal biosignal decoding.

Findings

Hierarchical Transformer with attention fusion outperforms baselines by over 10%.

Cross-modal interactions contribute about 30% of decision signals.

Attention-driven fusion enhances biosignal classification accuracy.

Abstract

Hand gestures are a primary output of the human motor system, yet the decoding of their neuromuscular signatures remains a bottleneck for basic neuroscience and assistive technologies such as prosthetics. Traditional human-machine interface pipelines rely on a single biosignal modality, but multimodal fusion can exploit complementary information from sensors. We systematically compare linear and attention-based fusion strategies across three architectures: a Multimodal MLP, a Multimodal Transformer, and a Hierarchical Transformer, evaluating performance on scenarios with unimodal and multimodal inputs. Experiments use two publicly available datasets: NinaPro DB2 (sEMG and accelerometer) and HD-sEMG 65-Gesture (high-density sEMG and force). Across both datasets, the Hierarchical Transformer with attention-based fusion consistently achieved the highest accuracy, surpassing the multimodal and best single-modality linear-fusion MLP baseline by over 10% on NinaPro DB2 and 3.7% on HD-sEMG. To investigate how modalities interact, we introduce an Isolation Network that selectively silences unimodal or cross-modal attention pathways, quantifying each group of token interactions' contribution to downstream decisions. Ablations reveal that cross-modal interactions contribute approximately 30% of the decision signal across transformer layers, highlighting the importance of attention-driven fusion in harnessing complementary modality information. Together, these findings reveal when and how multimodal fusion would enhance biosignal classification and also provides mechanistic insights of human muscle activities. The study would be beneficial in the design of sensor arrays for neurorobotic systems.