Fine-grained Soundscape Control for Augmented Hearing
Seunghyun Oh, Malek Itani, Aseem Gauri, Shyamnath Gollakota

TL;DR
Aurchestra is a novel system that enables real-time, fine-grained control over multiple overlapping sounds in augmented hearing devices, allowing users to independently adjust sound sources in complex environments.
Contribution
It introduces a resource-efficient, on-device multi-output extraction network and a dynamic interface for independent sound source control in hearables.
Findings
Supports up to 5 overlapping sounds with robust performance.
Achieves real-time processing on resource-limited hardware.
Significantly improves target sound enhancement and interference suppression.
Abstract
Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic scenes, however, contain many simultaneous sources that users may want to adjust independently. We introduce Aurchestra, the first system to provide fine-grained, real-time soundscape control on resource-constrained hearables. Our system has two key components: (1) a dynamic interface that surfaces only active sound classes and (2) a real-time, on-device multi-output extraction network that generates separate streams for each selected class, achieving robust performance for upto 5 overlapping target sounds, and letting users mix their environment by customizing per-class volumes, much like an audio engineer mixes tracks. We optimize the model architecture for multiple compute-limited platforms and demonstrate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHearing Loss and Rehabilitation · Speech and Audio Processing · Advanced Adaptive Filtering Techniques
