A quest through interconnected datasets: lessons from highly-cited ICASSP papers

TL;DR

This study analyzes highly-cited ICASSP papers to understand dataset origins and emphasizes the importance of transparency and accountability in data provenance for audio machine learning applications.

Contribution

It provides a detailed investigation into dataset usage in top ICASSP papers and advocates for greater transparency in data origins in the community.

Findings

Many datasets have unclear or entangled origins.

Current reporting often lacks detailed data provenance.

Community should incentivize explicit data documentation.

Abstract

As audio machine learning outcomes are deployed in societally impactful applications, it is important to have a sense of the quality and origins of the data used. Noticing that being explicit about this sense is not trivially rewarded in academic publishing in applied machine learning domains, and neither is included in typical applied machine learning curricula, we present a study into dataset usage connected to the top-5 cited papers at the International Conference on Acoustics, Speech, and Signal Processing (ICASSP). In this, we conduct thorough depth-first analyses towards origins of used datasets, often leading to searches that had to go beyond what was reported in official papers, and ending into unclear or entangled origins. Especially in the current pull towards larger, and possibly generative AI models, awareness of the need for accountability on data provenance is increasing. With this, we call on the community to not only focus on engineering larger models, but create more room and reward for explicitizing the foundations on which such models should be built.