Cross-Modal Binary Attention: An Energy-Efficient Fusion Framework for Audio-Visual Learning

TL;DR

This paper introduces CMQKA, a scalable, energy-efficient cross-modal fusion mechanism for audio-visual learning that enables hierarchical integration of features with state-of-the-art performance.

Contribution

The paper presents CMQKA, a novel linear-complexity attention mechanism, and SNNergy, a hierarchical, energy-efficient fusion framework utilizing event-driven binary operations.

Findings

Achieves linear O(N) complexity in cross-modal attention.

Outperforms existing methods on CREMA-D, AVE, and UrbanSound8K-AV benchmarks.

Maintains high fusion effectiveness with significant energy savings.

Abstract

Effective multimodal fusion requires mechanisms that can capture complex cross-modal dependencies while remaining computationally scalable for real-world deployment. Existing audio-visual fusion approaches face a fundamental trade-off: attention-based methods effectively model cross-modal relationships but incur quadratic computational complexity that prevents hierarchical, multi-scale architectures, while efficient fusion strategies rely on simplistic concatenation that fails to extract complementary cross-modal information. We introduce CMQKA, a novel cross-modal fusion mechanism that achieves linear O(N) complexity through efficient binary operations, enabling scalable hierarchical fusion previously infeasible with conventional attention. CMQKA employs bidirectional cross-modal Query-Key attention to extract complementary spatiotemporal features and uses learnable residual fusion to preserve modality-specific characteristics while enriching representations with cross-modal information. Building upon CMQKA, we present SNNergy, an energy-efficient multimodal fusion framework with a hierarchical architecture that processes inputs through progressively decreasing spatial resolutions and increasing semantic abstraction. This multi-scale fusion capability allows the framework to capture both local patterns and global context across modalities. Implemented with event-driven binary spike operations, SNNergy achieves remarkable energy efficiency while maintaining fusion effectiveness and establishing new state-of-the-art results on challenging audio-visual benchmarks, including CREMA-D, AVE, and UrbanSound8K-AV, significantly outperforming existing multimodal fusion baselines. Our framework advances multimodal fusion by introducing a scalable fusion mechanism that enables hierarchical cross-modal integration with practical energy efficiency for real-world audio-visual intelligence systems.