Learning Word Embeddings from Speech

TL;DR

This paper introduces Sequence-to-Sequence Audio2Vec, a neural network model that learns semantic audio segment embeddings directly from raw speech, enabling semantic analysis without text or image data.

Contribution

The paper presents a novel unsupervised neural architecture for extracting semantic audio embeddings directly from speech, bypassing the need for text or image annotations.

Findings

Achieved competitive word similarity scores compared to GloVe.

Demonstrated effective semantic representation of speech segments.

Model works directly on raw speech without additional modalities.

Abstract

In this paper, we propose a novel deep neural network architecture, Sequence-to-Sequence Audio2Vec, for unsupervised learning of fixed-length vector representations of audio segments excised from a speech corpus, where the vectors contain semantic information pertaining to the segments, and are close to other vectors in the embedding space if their corresponding segments are semantically similar. The design of the proposed model is based on the RNN Encoder-Decoder framework, and borrows the methodology of continuous skip-grams for training. The learned vector representations are evaluated on 13 widely used word similarity benchmarks, and achieved competitive results to that of GloVe. The biggest advantage of the proposed model is its capability of extracting semantic information of audio segments taken directly from raw speech, without relying on any other modalities such as text or images, which are challenging and expensive to collect and annotate.