Fairness and underspecification in acoustic scene classification: The case for disaggregated evaluations

TL;DR

This paper advocates for disaggregated evaluations in acoustic scene classification to uncover biases and fairness issues across factors like location and device, revealing significant biases in standard models.

Contribution

It introduces a holistic evaluation approach for ASC models that considers performance across various sub-populations, highlighting biases and underspecification.

Findings

All examined architectures show biases across factors.

Biases vary significantly between different architectures.

Disaggregated evaluation reveals fairness issues not seen in aggregate metrics.

Abstract

Underspecification and fairness in machine learning (ML) applications have recently become two prominent issues in the ML community. Acoustic scene classification (ASC) applications have so far remained unaffected by this discussion, but are now becoming increasingly used in real-world systems where fairness and reliability are critical aspects. In this work, we argue for the need of a more holistic evaluation process for ASC models through disaggregated evaluations. This entails taking into account performance differences across several factors, such as city, location, and recording device. Although these factors play a well-understood role in the performance of ASC models, most works report single evaluation metrics taking into account all different strata of a particular dataset. We argue that metrics computed on specific sub-populations of the underlying data contain valuable information about the expected real-world behaviour of proposed systems, and their reporting could improve the transparency and trustability of such systems. We demonstrate the effectiveness of the proposed evaluation process in uncovering underspecification and fairness problems exhibited by several standard ML architectures when trained on two widely-used ASC datasets. Our evaluation shows that all examined architectures exhibit large biases across all factors taken into consideration, and in particular with respect to the recording location. Additionally, different architectures exhibit different biases even though they are trained with the same experimental configurations.