Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning

TL;DR

This paper introduces a novel Transformer-based multimodal Spiking Neural Network framework called S-CMRL that effectively integrates audio-visual information through semantic alignment and residual learning, achieving state-of-the-art results.

Contribution

The paper proposes a new SNN architecture with semantic-alignment cross-modal residual learning for improved audio-visual integration in multimodal scenarios.

Findings

S-CMRL outperforms existing methods on benchmark datasets.

Achieves state-of-the-art performance in multimodal SNN tasks.

Demonstrates effective cross-modal feature alignment and fusion.

Abstract

Humans interpret and perceive the world by integrating sensory information from multiple modalities, such as vision and hearing. Spiking Neural Networks (SNNs), as brain-inspired computational models, exhibit unique advantages in emulating the brain's information processing mechanisms. However, existing SNN models primarily focus on unimodal processing and lack efficient cross-modal information fusion, thereby limiting their effectiveness in real-world multimodal scenarios. To address this challenge, we propose a semantic-alignment cross-modal residual learning (S-CMRL) framework, a Transformer-based multimodal SNN architecture designed for effective audio-visual integration. S-CMRL leverages a spatiotemporal spiking attention mechanism to extract complementary features across modalities, and incorporates a cross-modal residual learning strategy to enhance feature integration. Additionally, a semantic alignment optimization mechanism is introduced to align cross-modal features within a shared semantic space, improving their consistency and complementarity. Extensive experiments on three benchmark datasets CREMA-D, UrbanSound8K-AV, and MNISTDVS-NTIDIGITS demonstrate that S-CMRL significantly outperforms existing multimodal SNN methods, achieving the state-of-the-art performance. The code is publicly available at https://github.com/Brain-Cog-Lab/S-CMRL.