Characterisation of speech diversity using self-organising maps

TL;DR

This paper explores using self-organising maps, specifically Kohonen SOMs, to classify large unlabelled speech datasets with minimal phoneme annotations, focusing on Australian vowel pronunciation.

Contribution

It introduces a semi-supervised method employing multilevel SOMs for evaluating pronunciation, advancing speaker classification and vowel analysis in speech data.

Findings

Low phoneme error rates achieved

Effective evaluation of pronunciation using multilevel SOMs

Application to Australian vowel pronunciation

Abstract

We report investigations into speaker classification of larger quantities of unlabelled speech data using small sets of manually phonemically annotated speech. The Kohonen speech typewriter is a semi-supervised method comprised of self-organising maps (SOMs) that achieves low phoneme error rates. A SOM is a 2D array of cells that learn vector representations of the data based on neighbourhoods. In this paper, we report a method to evaluate pronunciation using multilevel SOMs with /hVd/ single syllable utterances for the study of vowels, for Australian pronunciation.

Full text

Characterisation of speech diversity using self-organising maps

Index Terms: Kohonen speech typewriter, Australian speech, semi-supervised SOM training and boosting

1 Introduction

We report investigations into speaker classification of larger quantities of unlabelled speech data using small sets of manually phonemically annotated speech. The Kohonen speech typewriter [1] is a semi-supervised method comprised of self-organising maps (SOMs) that achieves low phoneme error rates. A SOM is a 2D array of cells that learn vector representations of the data based on neighbourhoods. In this paper, we report a method to evaluate pronunciation using multilevel SOMs with /hVd/ single syllable utterances for the study of vowels, following [2] (for Australian pronunciation).

2 Methodology

We used 18 /hVd/ words from AusTalk, an Australian speech research corpus collected from speakers of Australian English [3]. Audio was converted to conventional 39-feature mel-frequency cepstral coefficients (MFCCs, log energy, and differential and acceleration calculated on 25 ms windows with 10 ms offsets).111MATLAB scripts are available from the authors on request. Scripts for MFCCs were modified from mathworks.com/matlabcentral/fileexchange/32849-htk-mfcc-matlab.

To investigate the sensitivity of our phonemic model to dialect, the system was trained using unlabelled speech from one of three different groups, General Australian (21 speakers), educated Melbournian (21 aged 25-34y), or speakers of Chinese (14), and labelled using a subset of manually annotated data (phonemes and start/end points) from three further speakers, as follows: a base 25x25 unit SOM was trained using all audio windows from the unlabelled data; three-quarters of the annotated data was assigned to the units in the base map; the resulting labelled map was used for separating the unlabelled input data for submap boosting. Two approaches to submapping, each with three 20x20 submap SOMs, were trained on data segmented by the base map by: (1) three maps for subsets of vowels with shared confusion in the base map (Kohonen's method); or (2) three maps of either /h/, vowels, or /d/ (linguistic).

3 Results

Frequently confused phonemes (on the first map) were identified using a greedy method based on high off-diagonal values in the confusion matrix from the base map:

• •

Vowel Group 1: /o/ (hoyd) and /o/ (horde)

• •

Vowel Group 2: /e/ (head), /e/ (haired), // (herd), // (hid), and // (heared)

• •

Vowel Group 3: // (hud), /æ/ (howd), /æ/ (hade), // (hard), /e/ (hide), // (hod), and // (hode)

The vowel error rate, shown in Table 1, was calculated by comparing the most frequent vowel label outputted for an audio file with the vowel noted by manual annotators. Our method of using initial, vowel, or final for submaps was observed to be slightly better than Kohonen's method of using submaps to disambiguate confused phonemes, and particularly for Chinese, marginally significant.

4 Discussion

The goal of the experiments conducted in this work was to explore the general capacity of labelled data for revealing characteristics of dialect in the unlabelled speech of Chinese-background Australians and educated Melbournians vs Australians generally. The results are competitive with the original phonetic typewriter results, demonstrating effectiveness with 39-feature MFCC vectors. There is the possibility for improving on these using context, but for the future directions of this research, we are more interested in characterising differences between the three groups.

5 Acknowledgements

The AusTalk corpus was collected as part of the Big ASC project funded by the Australian Research Council (LE100100211).222See https://austalk.edu.au/bibliography.html for details.