Contextualized Spoken Word Representations from Convolutional Autoencoders

TL;DR

This paper introduces a convolutional autoencoder model to generate contextualized spoken word representations, aiming to improve audio-based NLP tasks by capturing syntactic, semantic, and paralinguistic features.

Contribution

It presents a novel neural architecture for creating context-aware spoken word embeddings that outperform existing text-based models on benchmark datasets.

Findings

The model produces robust, meaningful vector spaces for spoken words.

It outperforms traditional text-based models on word similarity tasks.

The approach preserves tone, expression, and accent information.

Abstract

A lot of work has been done to build text-based language models for performing different NLP tasks, but not much research has been done in the case of audio-based language models. This paper proposes a Convolutional Autoencoder based neural architecture to model syntactically and semantically adequate contextualized representations of varying length spoken words. The use of such representations can not only lead to great advances in the audio-based NLP tasks but can also curtail the loss of information like tone, expression, accent, etc while converting speech to text to perform these tasks. The performance of the proposed model is validated by (1) examining the generated vector space, and (2) evaluating its performance on three benchmark datasets for measuring word similarities, against existing widely used text-based language models that are trained on the transcriptions. The proposed model was able to demonstrate its robustness when compared to the other two language-based models.