MASR: Multi-label Aware Speech Representation

TL;DR

MASR introduces a multi-label aware framework for speech representation learning that incorporates external knowledge sources, improving performance on various downstream tasks like language identification and speech recognition.

Contribution

It proposes a novel framework that integrates external meta-data into SSL-based speech representations, enhancing their effectiveness across multiple tasks.

Findings

Significant performance improvements over benchmarks.

Enhanced separation of closely related languages.

Versatile integration with any SSL method.

Abstract

In the recent years, speech representation learning is constructed primarily as a self-supervised learning (SSL) task, using the raw audio signal alone, while ignoring the side-information that is often available for a given speech recording. In this paper, we propose MASR, a Multi-label Aware Speech Representation learning framework, which addresses the aforementioned limitations. MASR enables the inclusion of multiple external knowledge sources to enhance the utilization of meta-data information. The external knowledge sources are incorporated in the form of sample-level pair-wise similarity matrices that are useful in a hard-mining loss. A key advantage of the MASR framework is that it can be combined with any choice of SSL method. Using MASR representations, we perform evaluations on several downstream tasks such as language identification, speech recognition and other non-semantic tasks such as speaker and emotion recognition. In these experiments, we illustrate significant performance improvements for the MASR over other established benchmarks. We perform a detailed analysis on the language identification task to provide insights on how the proposed loss function enables the representations to separate closely related languages.