Modality-Specific Speech Enhancement and Noise-Adaptive Fusion for Acoustic and Body-Conduction Microphone Framework

TL;DR

This paper introduces a multi-modal speech enhancement framework combining body-conduction and acoustic microphones, employing specialized networks and dynamic fusion to improve noise suppression and high-frequency reconstruction in noisy environments.

Contribution

It presents a novel multi-modal framework with specialized networks and adaptive fusion, advancing noise suppression and high-frequency recovery over traditional single-modal methods.

Findings

Outperforms single-modal solutions in noisy environments

Effective dynamic fusion adapts to local noise conditions

Improves high-frequency reconstruction and noise suppression

Abstract

Body-conduction microphone signals (BMS) bypass airborne sound, providing strong noise resistance. However, a complementary modality is required to compensate for the inherent loss of high-frequency information. In this study, we propose a novel multi-modal framework that combines BMS and acoustic microphone signals (AMS) to achieve both noise suppression and high-frequency reconstruction. Unlike conventional multi-modal approaches that simply merge features, our method employs two specialized networks: a mapping-based model to enhance BMS and a masking-based model to denoise AMS. These networks are integrated through a dynamic fusion mechanism that adapts to local noise conditions, ensuring the optimal use of each modality's strengths. We performed evaluations on the TAPS dataset, augmented with DNS-2023 noise clips, using objective speech quality metrics. The results clearly demonstrate that our approach outperforms single-modal solutions in a wide range of noisy environments.