Deep factorization for speech signal

TL;DR

This paper introduces a cascade deep factorization framework that sequentially infers and separates multiple informative factors in speech signals using deep neural networks, enabling accurate speech spectrum reconstruction and benefiting tasks like speaker and emotion recognition.

Contribution

The paper proposes a novel cascade deep factorization framework that effectively separates speech factors and reconstructs speech signals, demonstrating its utility in various speech processing applications.

Findings

Effective factorization of speech signals into multiple factors.

High-accuracy reconstruction of speech spectrum from inferred factors.

Potential improvements in speaker and emotion recognition tasks.

Abstract

Various informative factors mixed in speech signals, leading to great difficulty when decoding any of the factors. An intuitive idea is to factorize each speech frame into individual informative factors, though it turns out to be highly difficult. Recently, we found that speaker traits, which were assumed to be long-term distributional properties, are actually short-time patterns, and can be learned by a carefully designed deep neural network (DNN). This discovery motivated a cascade deep factorization (CDF) framework that will be presented in this paper. The proposed framework infers speech factors in a sequential way, where factors previously inferred are used as conditional variables when inferring other factors. We will show that this approach can effectively factorize speech signals, and using these factors, the original speech spectrum can be recovered with a high accuracy. This factorization and reconstruction approach provides potential values for many speech processing tasks, e.g., speaker recognition and emotion recognition, as will be demonstrated in the paper.