Synthia's Melody: A Benchmark Framework for Unsupervised Domain Adaptation in Audio

TL;DR

This paper introduces Synthia's melody, a versatile benchmark framework for evaluating unsupervised domain adaptation in audio, addressing the lack of suitable datasets and enabling controlled experiments on distribution shifts.

Contribution

The paper presents Synthia's melody, a novel audio data generation framework that creates unbiased, customizable melodies for benchmarking domain adaptation in audio deep learning models.

Findings

Synthia's melody enables controlled testing of distribution shifts in audio models.

Models show varying robustness under different domain shifts generated by the framework.

The benchmark facilitates reproducible and comparable experiments in audio domain adaptation.

Abstract

Despite significant advancements in deep learning for vision and natural language, unsupervised domain adaptation in audio remains relatively unexplored. We, in part, attribute this to the lack of an appropriate benchmark dataset. To address this gap, we present Synthia's melody, a novel audio data generation framework capable of simulating an infinite variety of 4-second melodies with user-specified confounding structures characterised by musical keys, timbre, and loudness. Unlike existing datasets collected under observational settings, Synthia's melody is free of unobserved biases, ensuring the reproducibility and comparability of experiments. To showcase its utility, we generate two types of distribution shifts-domain shift and sample selection bias-and evaluate the performance of acoustic deep learning models under these shifts. Our evaluations reveal that Synthia's melody provides a robust testbed for examining the susceptibility of these models to varying levels of distribution shift.