Spatio-temporal Latent Representations for the Analysis of Acoustic Scenes in-the-wild

TL;DR

This paper introduces a novel self-supervised learning approach to extract spatio-temporal acoustic scene representations from in-the-wild audio data, enabling differentiation of environments like indoor and subway settings.

Contribution

It presents a new method combining acoustic embeddings, NLP-inspired algorithms, and VAEs to characterize and visualize acoustic scenes in real-world data.

Findings

Distinct acoustic scenes can be identified in the latent space.

Indoor and subway environments show clear separation in the embeddings.

The approach effectively captures spatio-temporal variations in in-the-wild audio.

Abstract

In the field of acoustic scene analysis, this paper presents a novel approach to find spatio-temporal latent representations from in-the-wild audio data. By using WE-LIVE, an in-house collected dataset that includes audio recordings in diverse real-world environments together with sparse GPS coordinates, self-annotated emotional and situational labels, we tackle the challenging task of associating each audio segment with its corresponding location as a pretext task, with the final aim of acoustically detecting violent (anomalous) contexts, left as further work. By generating acoustic embeddings and using the self-supervised learning paradigm, we aim to use the model-generated latent space to acoustically characterize the spatio-temporal context. We use YAMNet, an acoustic events classifier trained in AudioSet to temporally locate and identify acoustic events in WE-LIVE. In order to transform the discrete acoustic events into embeddings, we compare the information-retrieval-based TF-IDF algorithm and Node2Vec as an analogy to Natural Language Processing techniques. A VAE is then trained to provide a further adapted latent space. The analysis was carried out by measuring the cosine distance and visualizing data distribution via t-Distributed Stochastic Neighbor Embedding, revealing distinct acoustic scenes. Specifically, we discern variations between indoor and subway environments. Notably, these distinctions emerge within the latent space of the VAE, a stark contrast to the random distribution of data points before encoding. In summary, our research contributes a pioneering approach for extracting spatio-temporal latent representations from in-the-wild audio data.