Proactive Hearing Assistants that Isolate Egocentric Conversations
Guilin Hu, Malek Itani, Tuochao Chen, Shyamnath Gollakota

TL;DR
This paper presents a real-time, on-device proactive hearing assistant that isolates conversation partners using egocentric binaural audio and self-speech cues, improving multi-party conversation understanding.
Contribution
It introduces a dual-model architecture for real-time partner identification and isolation in egocentric audio, advancing proactive hearing assistance technology.
Findings
Effective partner identification in multi-party conversations
Generalizes well across different speakers and settings
Operates with low latency on-device
Abstract
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsSocial Robot Interaction and HRI · AI in Service Interactions · Emotion and Mood Recognition
