Learning Audio Representations with MLPs

TL;DR

This paper introduces a simple yet effective MLP-based method for learning audio representations, achieving top performance and high efficiency on multiple benchmarks in the HEAR challenge.

Contribution

It presents a novel MLP-based approach for audio embedding generation and demonstrates its effectiveness and efficiency across diverse audio benchmarks.

Findings

Achieved first place in multiple HEAR challenge benchmarks.

Outperformed other methods in resource efficiency.

Produced high-quality timestamp and scene-level embeddings.

Abstract

In this paper, we propose an efficient MLP-based approach for learning audio representations, namely timestamp and scene-level audio embeddings. We use an encoder consisting of sequentially stacked gated MLP blocks, which accept 2D MFCCs as inputs. In addition, we also provide a simple temporal interpolation-based algorithm for computing scene-level embeddings from timestamp embeddings. The audio representations generated by our method are evaluated across a diverse set of benchmarks at the Holistic Evaluation of Audio Representations (HEAR) challenge, hosted at the NeurIPS 2021 competition track. We achieved first place on the Speech Commands (full), Speech Commands (5 hours), and the Mridingham Tonic benchmarks. Furthermore, our approach is also the most resource-efficient among all the submitted methods, in terms of both the number of model parameters and the time required to compute embeddings.