Deep Factorization for Speech Signal

TL;DR

This paper introduces a deep neural network-based cascade framework that effectively factorizes speech signals into independent factors, including speaker traits, enabling accurate speech spectrum reconstruction and advancing speech processing techniques.

Contribution

The paper presents a novel cascade deep factorization framework that sequentially infers speech factors, including short-time speaker traits, using a simple DNN, which was previously thought to be long-term only.

Findings

Speaker traits can be identified from a few frames using DNN.

The proposed method effectively factorizes speech signals in an AER task.

Speech spectrum can be reconstructed with high accuracy from inferred factors.

Abstract

Speech signals are complex intermingling of various informative factors, and this information blending makes decoding any of the individual factors extremely difficult. A natural idea is to factorize each speech frame into independent factors, though it turns out to be even more difficult than decoding each individual factor. A major encumbrance is that the speaker trait, a major factor in speech signals, has been suspected to be a long-term distributional pattern and so not identifiable at the frame level. In this paper, we demonstrated that the speaker factor is also a short-time spectral pattern and can be largely identified with just a few frames using a simple deep neural network (DNN). This discovery motivated a cascade deep factorization (CDF) framework that infers speech factors in a sequential way, and factors previously inferred are used as conditional variables when inferring other factors. Our experiment on an automatic emotion recognition (AER) task demonstrated that this approach can effectively factorize speech signals, and using these factors, the original speech spectrum can be recovered with high accuracy. This factorization and reconstruction approach provides a novel tool for many speech processing tasks.