Supervised Classifiers for Audio Impairments with Noisy Labels

TL;DR

This paper investigates how supervised audio impairment classifiers trained on noisy user feedback labels perform, demonstrating CNNs' robustness to label noise and the need for larger datasets with increased noise levels.

Contribution

It provides an analysis of CNN performance on noisy labels in audio impairment classification and highlights the importance of larger datasets for noisy label training.

Findings

CNNs outperform dense networks on noisy labels

Training with noisy labels requires larger datasets

CNNs generalize better with noisy label data

Abstract

Voice-over-Internet-Protocol (VoIP) calls are prone to various speech impairments due to environmental and network conditions resulting in bad user experience. A reliable audio impairment classifier helps to identify the cause for bad audio quality. The user feedback after the call can act as the ground truth labels for training a supervised classifier on a large audio dataset. However, the labels are noisy as most of the users lack the expertise to precisely articulate the impairment in the perceived speech. In this paper, we analyze the effects of massive noise in labels in training dense networks and Convolutional Neural Networks (CNN) using engineered features, spectrograms and raw audio samples as inputs. We demonstrate that CNN can generalize better on the training data with a large number of noisy labels and gives remarkably higher test performance. The classifiers were trained both on randomly generated label noise and the label noise introduced by human errors. We also show that training with noisy labels requires a significant increase in the training dataset size, which is in proportion to the amount of noise in the labels.