A Joint Spectro-Temporal Relational Thinking Based Acoustic Modeling Framework

TL;DR

This paper introduces a novel spectro-temporal relational thinking framework for acoustic modeling in speech recognition, leveraging probabilistic graphs to improve phoneme recognition accuracy, especially for vowels.

Contribution

It presents a new framework that models speech relations across time and frequency domains using probabilistic graphs, enhancing recognition performance.

Findings

Achieved 7.82% improvement in phoneme recognition on TIMIT.

Model particularly improves vowel recognition accuracy.

Relational modeling captures complex speech dependencies.

Abstract

Relational thinking refers to the inherent ability of humans to form mental impressions about relations between sensory signals and prior knowledge, and subsequently incorporate them into their model of their world. Despite the crucial role relational thinking plays in human understanding of speech, it has yet to be leveraged in any artificial speech recognition systems. Recently, there have been some attempts to correct this oversight, but these have been limited to coarse utterance-level models that operate exclusively in the time domain. In an attempt to narrow the gap between artificial systems and human abilities, this paper presents a novel spectro-temporal relational thinking based acoustic modeling framework. Specifically, it first generates numerous probabilistic graphs to model the relationships among speech segments across both time and frequency domains. The relational information rooted in every pair of nodes within these graphs is then aggregated and embedded into latent representations that can be utilized by downstream tasks. Models built upon this framework outperform state-of-the-art systems with a 7.82\% improvement in phoneme recognition tasks over the TIMIT dataset. In-depth analyses further reveal that our proposed relational thinking modeling mainly improves the model's ability to recognize vowels, which are the most likely to be confused by phoneme recognizers.