A Multi-layered Acoustic Tokenizing Deep Neural Network (MAT-DNN) for Unsupervised Discovery of Linguistic Units and Generation of High Quality Features

TL;DR

This paper introduces MAT-DNN, an iterative deep learning system that automatically discovers linguistic units from unlabeled speech and generates high-quality features, advancing zero-resource speech processing.

Contribution

The paper presents a novel multi-layered acoustic tokenizing deep neural network that jointly discovers multiple sets of acoustic tokens and extracts high-quality features through an iterative feedback loop.

Findings

Successfully participated in the Zero Resource Speech Challenge

Discovered multiple sets of acoustic tokens with different characteristics

Generated high-quality features for speech processing

Abstract

This paper summarizes the work done by the authors for the Zero Resource Speech Challenge organized in the technical program of Interspeech 2015. The goal of the challenge is to discover linguistic units directly from unlabeled speech data. The Multi-layered Acoustic Tokenizer (MAT) proposed in this work automatically discovers multiple sets of acoustic tokens from the given corpus. Each acoustic token set is specified by a set of hyperparameters that describe the model configuration. These sets of acoustic tokens carry different characteristics of the given corpus and the language behind thus can be mutually reinforced. The multiple sets of token labels are then used as the targets of a Multi-target DNN (MDNN) trained on low-level acoustic features. Bottleneck features extracted from the MDNN are used as feedback for the MAT and the MDNN itself. We call this iterative system the Multi-layered Acoustic Tokenizing Deep Neural Network (MAT-DNN) which generates high quality features for track 1 of the challenge and acoustic tokens for track 2 of the challenge.