Perceptually Aligning Representations of Music via Noise-Augmented Autoencoders

TL;DR

This paper introduces a novel autoencoder training method combining noise augmentation and perceptual losses to create hierarchical, perceptually aligned audio representations that enhance music analysis and brain response prediction.

Contribution

It presents a new training approach for autoencoders that produces perceptually structured audio embeddings, improving music and brain response modeling.

Findings

Hierarchical structure emerges in audio representations.

Perceptual hierarchies improve music surprisal estimation.

Enhanced prediction of EEG responses to music.

Abstract

We argue that training autoencoders to reconstruct inputs from noised versions of their encodings, when combined with perceptual losses, yields encodings that are structured according to a perceptual hierarchy. We demonstrate the emergence of this hierarchical structure by showing that, after training an audio autoencoder in this manner, perceptually salient information is captured in coarser representation structures than with conventional training. Furthermore, we show that such perceptual hierarchies improve latent diffusion decoding in the context of estimating surprisal in music pitches and predicting EEG-brain responses to music listening. Pretrained weights are available on github.com/CPJKU/pa-audioic.