If It's Good Enough for You, It's Good Enough for Me: Transferability of Audio Sufficiencies across Models

TL;DR

This paper introduces transferability analysis to study how minimal, sufficient audio signals for classification transfer across different models, revealing task-dependent transferability rates and model-specific behaviors.

Contribution

It defines transferability of sufficient signals and conducts a large study across three audio classification tasks, uncovering task-specific transferability patterns and model differences.

Findings

Music genre transferability rate is approximately 26%.

Deepfake detection models show high variance in transferability.

Transferability analysis uncovers information-theoretic differences beyond accuracy.

Abstract

In order to gain fresh insights about the information processing characteristics of different audio classification models, we propose transferability analysis. Given a minimal, sufficient signal for a classification on a model $f$, transferability analysis asks whether other models accept this minimal signal as having the same classification as it did on $f$. We define what it means for a sufficient signal to be transferable and perform a large study over $3$ different classification tasks: music genre, emotion recognition and deepfake detection. We find that transferability rates vary depending on the task, with sufficient signals for music genre being transferable $\approx26\%$ of the time. The other tasks reveal much higher variance in transferability and reveal that some models, in particular on deepfake detection, have different transferability behavior. We call these models `flat-earther' models. We investigate deepfake audio in more depth, and show that transferability analysis also allows to us to discover information theoretic differences between the models which are not captured by the more familiar metrics of accuracy and precision.