Fine-grained Soundscape Control for Augmented Hearing

TL;DR

Aurchestra is a novel system that enables real-time, fine-grained control over multiple overlapping sounds in augmented hearing devices, allowing users to independently adjust sound sources in complex environments.

Contribution

It introduces a resource-efficient, on-device multi-output extraction network and a dynamic interface for independent sound source control in hearables.

Findings

Supports up to 5 overlapping sounds with robust performance.

Achieves real-time processing on resource-limited hardware.

Significantly improves target sound enhancement and interference suppression.

Abstract

Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic scenes, however, contain many simultaneous sources that users may want to adjust independently. We introduce Aurchestra, the first system to provide fine-grained, real-time soundscape control on resource-constrained hearables. Our system has two key components: (1) a dynamic interface that surfaces only active sound classes and (2) a real-time, on-device multi-output extraction network that generates separate streams for each selected class, achieving robust performance for upto 5 overlapping target sounds, and letting users mix their environment by customizing per-class volumes, much like an audio engineer mixes tracks. We optimize the model architecture for multiple compute-limited platforms and demonstrate real-time performance on 6 ms streaming audio chunks. Across real-world environments in previously unseen indoor and outdoor scenarios, our system enables expressive per-class sound control and achieves substantial improvements in target-class enhancement and interference suppression. Our results show that the world need not be heard as a single, undifferentiated stream: with Aurchestra, the soundscape becomes truly programmable.