Understanding Audio Features via Trainable Basis Functions

TL;DR

This paper investigates making spectrogram basis functions trainable to enhance audio feature representation, demonstrating significant improvements in keyword spotting and speech recognition accuracy, especially in models with limited complexity.

Contribution

It introduces trainable basis functions for spectrograms, allowing models to adapt features for specific tasks, which improves performance in KWS and ASR.

Findings

Boosted KWS accuracy by 14.2 percentage points

Lowered Phone Error Rate by 9.5 percentage points

Provided insights into important frequency bins for tasks

Abstract

In this paper we explore the possibility of maximizing the information represented in spectrograms by making the spectrogram basis functions trainable. We experiment with two different tasks, namely keyword spotting (KWS) and automatic speech recognition (ASR). For most neural network models, the architecture and hyperparameters are typically fine-tuned and optimized in experiments. Input features, however, are often treated as fixed. In the case of audio, signals can be mainly expressed in two main ways: raw waveforms (time-domain) or spectrograms (time-frequency-domain). In addition, different spectrogram types are often used and tailored to fit different applications. In our experiments, we allow for this tailoring directly as part of the network. Our experimental results show that using trainable basis functions can boost the accuracy of Keyword Spotting (KWS) by 14.2 percentage points, and lower the Phone Error Rate (PER) by 9.5 percentage points. Although models using trainable basis functions become less effective as the model complexity increases, the trained filter shapes could still provide us with insights on which frequency bins are important for that specific task. From our experiments, we can conclude that trainable basis functions are a useful tool to boost the performance when the model complexity is limited.