Residual Learning for Neural Ambisonics Encoders

TL;DR

This paper proposes a residual learning framework that combines linear and neural network encoders to improve spatial audio capture for wearable devices, demonstrating consistent improvements in real-world scenarios.

Contribution

It introduces a residual learning approach that refines linear Ambisonics encoders with neural networks, enhancing performance in practical applications.

Findings

Neural encoders outperform linear baseline only within residual framework.

Residual models show significant improvements across all metrics for in-domain data.

Neural encoders still struggle with high-frequency directional accuracy.

Abstract

Emerging wearable devices such as smartglasses and extended reality headsets demand high-quality spatial audio capture from compact, head-worn microphone arrays. Ambisonics provides a device-agnostic spatial audio representation by mapping array signals to spherical harmonic (SH) coefficients. In practice, however, accurate encoding remains challenging. While traditional linear encoders are signal-independent and robust, they amplify low-frequency noise and suffer from high-frequency spatial aliasing. On the other hand, neural network approaches can outperform linear encoders but they often assume idealized microphones and may perform inconsistently in real-world scenarios. To leverage their complementary strengths, we introduce a residual-learning framework that refines a linear encoder with corrections from a neural network. Using measured array transfer functions from smartglasses, we compare a UNet-based encoder from the literature with a new recurrent attention model. Our analysis reveals that both neural encoders only consistently outperform the linear baseline when integrated within the residual learning framework. In the residual configuration, both neural models achieve consistent and significant improvements across all tested metrics for in-domain data and moderate gains for out-of-domain data. Yet, coherence analysis indicates that all neural encoder configurations continue to struggle with directionally accurate high-frequency encoding.