Multimodal Input Aids a Bayesian Model of Phonetic Learning

TL;DR

This paper demonstrates that integrating visual facial cues with audio significantly improves phonetic discrimination in computational models, especially in noisy environments, suggesting visual information aids language learning.

Contribution

It introduces a method for creating synthetic videos for audiovisual phonetic learning models and shows that visual cues enhance phoneme discrimination performance.

Findings

Audiovisual models outperform audio-only models by up to 8.1% in phoneme discrimination.

Visual information helps maintain performance in noisy conditions, reducing the impact of noise.

Synthetic videos of speakers' faces improve the training and testing of audiovisual speech models.

Abstract

One of the many tasks facing the typically-developing child language learner is learning to discriminate between the distinctive sounds that make up words in their native language. Here we investigate whether multimodal information--specifically adult speech coupled with video frames of speakers' faces--benefits a computational model of phonetic learning. We introduce a method for creating high-quality synthetic videos of speakers' faces for an existing audio corpus. Our learning model, when both trained and tested on audiovisual inputs, achieves up to a 8.1% relative improvement on a phoneme discrimination battery compared to a model trained and tested on audio-only input. It also outperforms the audio model by up to 3.9% when both are tested on audio-only data, suggesting that visual information facilitates the acquisition of acoustic distinctions. Visual information is especially beneficial in noisy audio environments, where an audiovisual model closes 67% of the loss in discrimination performance of the audio model in noise relative to a non-noisy environment. These results demonstrate that visual information benefits an ideal learner and illustrate some of the ways that children might be able to leverage visual cues when learning to discriminate speech sounds.