FusionEnsemble-Net: An Attention-Based Ensemble of Spatiotemporal Networks for Multimodal Sign Language Recognition

TL;DR

FusionEnsemble-Net introduces an attention-based ensemble of spatiotemporal networks that effectively fuses visual and motion data, achieving high accuracy in multimodal sign language recognition, especially for complex gestures.

Contribution

It presents a novel ensemble framework with attention-based fusion of multimodal data, improving recognition accuracy over existing methods.

Findings

Achieved 99.44% accuracy on MultiMeDaLIS dataset.

Outperforms state-of-the-art sign language recognition approaches.

Demonstrates robustness through ensemble of diverse spatiotemporal networks.

Abstract

Accurate recognition of sign language in healthcare communication poses a significant challenge, requiring frameworks that can accurately interpret complex multimodal gestures. To deal with this, we propose FusionEnsemble-Net, a novel attention-based ensemble of spatiotemporal networks that dynamically fuses visual and motion data to enhance recognition accuracy. The proposed approach processes RGB video and range Doppler map radar modalities synchronously through four different spatiotemporal networks. For each network, features from both modalities are continuously fused using an attention-based fusion module before being fed into an ensemble of classifiers. Finally, the outputs of these four different fused channels are combined in an ensemble classification head, thereby enhancing the model's robustness. Experiments demonstrate that FusionEnsemble-Net outperforms state-of-the-art approaches with a test accuracy of 99.44% on the large-scale MultiMeDaLIS dataset for Italian Sign Language. Our findings indicate that an ensemble of diverse spatiotemporal networks, unified by attention-based fusion, yields a robust and accurate framework for complex, multimodal isolated gesture recognition tasks. The source code is available at: https://github.com/rezwanh001/Multimodal-Isolated-Italian-Sign-Language-Recognition.