Artificial Neural Networks Trained on Noisy Speech Exhibit the McGurk Effect

TL;DR

Artificial neural networks trained on audiovisual speech, even with only congruent data, can exhibit the McGurk effect, and noise during training influences the extent of audiovisual integration, providing insights into perception modeling.

Contribution

This study demonstrates that neural networks trained solely on congruent speech can produce the McGurk effect and explores how noisy training enhances audiovisual integration.

Findings

Networks trained on noisy speech show increased McGurk responses.

Increasing auditory noise during training initially boosts audiovisual integration.

Extreme noise levels impair the development of audiovisual speech integration.

Abstract

Humans are able to fuse information from both auditory and visual modalities to help with understanding speech. This is demonstrated through a phenomenon known as the McGurk Effect, during which a listener is presented with incongruent auditory and visual speech that fuse together into the percept of illusory intermediate phonemes. Building on a recent framework that proposes how to address developmental 'why' questions using artificial neural networks, we evaluated a set of recent artificial neural networks trained on audiovisual speech by testing them with audiovisually incongruent words designed to elicit the McGurk effect. We show that networks trained entirely on congruent audiovisual speech nevertheless exhibit the McGurk percept. We further investigated 'why' by comparing networks trained on clean speech to those trained on noisy speech, and discovered that training with noisy speech led to a pronounced increase in both visual responses and McGurk responses across all models. Furthermore, we observed that systematically increasing the level of auditory noise during ANN training also increased the amount of audiovisual integration up to a point, but at extreme noise levels, this integration failed to develop. These results suggest that excessive noise exposure during critical periods of audiovisual learning may negatively influence the development of audiovisual speech integration. This work also demonstrates that the McGurk effect reliably emerges untrained from the behaviour of both supervised and unsupervised networks, even networks trained only on congruent speech. This supports the notion that artificial neural networks might be useful models for certain aspects of perception and cognition.