Learning Joint Acoustic-Phonetic Word Embeddings

TL;DR

This paper introduces a method to learn shared acoustic-phonetic word embeddings using weak supervision, enabling the model to determine if two words sound similar across modalities with high accuracy.

Contribution

It proposes a novel joint embedding approach for acoustic and phonetic sequences, utilizing contrastive loss and hard negative mining, with strong empirical results.

Findings

Achieved 0.95 F1 score in word matching task

Demonstrated effectiveness of contrastive loss and negative mining techniques

Established a shared latent space for acoustic and phonetic word representations

Abstract

Most speech recognition tasks pertain to mapping words across two modalities: acoustic and orthographic. In this work, we suggest learning encoders that map variable-length, acoustic or phonetic, sequences that represent words into fixed-dimensional vectors in a shared latent space; such that the distance between two word vectors represents how closely the two words sound. Instead of directly learning the distances between word vectors, we employ weak supervision and model a binary classification task to predict whether two inputs, one of each modality, represent the same word given a distance threshold. We explore various deep-learning models, bimodal contrastive losses, and techniques for mining hard negative examples such as the semi-supervised technique of self-labeling. Our best model achieves an $F_1$ score of 0.95 for the binary classification task.