CoHear: Conversation Enhancement via Multi-Earphone Collaboration

TL;DR

ClearSphere is a system that enhances speech clarity in noisy, crowded environments by enabling multi-earphone collaboration, using a novel network protocol and deep learning for real-time, conversation-level speech extraction.

Contribution

It introduces a conversation-driven network protocol and a robust speech extraction model that work together for real-time, multi-user conversation enhancement without infrastructure.

Findings

Achieves over 90% accuracy in group formation

Improves speech quality by up to 8.8 dB over baselines

Demonstrates real-time performance on mobile devices

Abstract

In crowded places such as conferences, background noise, overlapping voices, and lively interactions make it difficult to have clear conversations. This situation often worsens the phenomenon known as "cocktail party deafness." We present ClearSphere, the collaborative system that enhances speech at the conversation level with multi-earphones. Real-time conversation enhancement requires a holistic modeling of all the members in the conversation, and an effective way to extract the speech from the mixture. ClearSphere bridges the acoustic sensor system and state-of-the-art deep learning for target speech extraction by making two key contributions: 1) a conversation-driven network protocol, and 2) a robust target conversation extraction model. Our networking protocol enables mobile, infrastructure-free coordination among earphone devices. Our conversation extraction model can leverage the relay audio in a bandwidth-efficient way. ClearSphere is evaluated in both real-world experiments and simulations. Results show that our conversation network obtains more than 90\% accuracy in group formation, improves the speech quality by up to 8.8 dB over state-of-the-art baselines, and demonstrates real-time performance on a mobile device. In a user study with 20 participants, ClearSphere has a much higher score than baseline with good usability.