DBMIF: a deep balanced multimodal iterative fusion framework for air- and bone-conduction speech enhancement

TL;DR

This paper introduces DBMIF, a novel multimodal speech enhancement framework that effectively combines air- and bone-conduction signals through iterative fusion, significantly improving speech quality and ASR performance in noisy environments.

Contribution

The paper presents a new three-branch deep fusion architecture with iterative cross-modal interaction and a balanced representation learning mechanism for robust speech enhancement.

Findings

DBMIF outperforms recent baselines in speech quality and intelligibility.

It reduces character error rate in ASR by at least 2.5%.

Demonstrates robustness across diverse noise types.

Abstract

The performance of conventional speech enhancement systems degrades sharply in extremely low signal-to-noise ratio (SNR) environments where air-conduction (AC) microphones are overwhelmed by ambient noise. Although bone-conduction (BC) sensors offer complementary, noise-tolerant information, existing fusion approaches struggle to maintain consistent performance across a wide range of SNR conditions. To address this limitation, we propose the Deep Balanced Multimodal Iterative Fusion Framework (DBMIF), a three-branch architecture designed to reconstruct high-fidelity speech through rigorous cross-modal interaction. Specifically, grounded in a multi-scale interactive encoder-decoder backbone, the framework orchestrates an iterative attention module and a cross-branch gated module to facilitate adaptive weighting and bidirectional exchange. To complement this dynamic interaction, a balanced-interaction bottleneck is further integrated to learn a compact, stable fused representation. Extensive experiments demonstrate that DBMIF achieves competitive performance compared with recent unimodal and multimodal baselines in both speech quality and intelligibility across diverse noise types. In downstream ASR tasks, the proposed method reduces the character error rate by at least 2.5 percent compared to competing approaches. These results confirm that DBMIF effectively harnesses the robustness of BC speech while preserving the naturalness of AC speech, ensuring reliability in real-world scenarios. The source code is publicly available at github.com/wyl516w/dbmif.