STEPs-RL: Speech-Text Entanglement for Phonetically Sound Representation Learning

TL;DR

This paper introduces STEPs-RL, a novel multi-modal neural network that combines speech and text entanglement to learn phonetically sound spoken-word representations, achieving high phonetic prediction accuracy and competitive word similarity results.

Contribution

It is the first work to use speech and text entanglement for learning spoken-word representations, integrating semantic, syntactic, and phonetic information.

Findings

Achieved 89.47% phonetic sequence prediction accuracy.

Outperformed traditional text-based models on word similarity benchmarks.

Captured phonetic structure effectively in the learned vector space.

Abstract

In this paper, we present a novel multi-modal deep neural network architecture that uses speech and text entanglement for learning phonetically sound spoken-word representations. STEPs-RL is trained in a supervised manner to predict the phonetic sequence of a target spoken-word using its contextual spoken word's speech and text, such that the model encodes its meaningful latent representations. Unlike existing work, we have used text along with speech for auditory representation learning to capture semantical and syntactical information along with the acoustic and temporal information. The latent representations produced by our model were not only able to predict the target phonetic sequences with an accuracy of 89.47% but were also able to achieve competitive results to textual word representation models, Word2Vec & FastText (trained on textual transcripts), when evaluated on four widely used word similarity benchmark datasets. In addition, investigation of the generated vector space also demonstrated the capability of the proposed model to capture the phonetic structure of the spoken-words. To the best of our knowledge, none of the existing works use speech and text entanglement for learning spoken-word representation, which makes this work first of its kind.