# A deep neural network model of audiovisual speech recognition reports the McGurk effect

## TL;DR

A deep neural network model called AVHuBERT shows a McGurk effect similar to humans, suggesting it can help understand how audio and visual speech cues are combined in perception.

## Contribution

AVHuBERT demonstrates a human-like McGurk effect, offering a new modeling framework for audiovisual speech perception.

## Key findings

- AVHuBERT identified McGurk syllables as non-“ba” on 59% of trials, similar to human responses.
- The model and humans produced a wide range of percepts, including “da,” “ah,” and “fa.”
- DNNs may complement Bayesian models in understanding audiovisual speech perception.

## Abstract

In the McGurk effect, perception of an auditory syllable changes dramatically when it is paired with an incongruent visual syllable, countering our intuition that speech perception is solely an auditory process. The dominant modeling framework for the study of audiovisual speech perception is that of Bayesian causal inference, but current Bayesian models are unable to predict the wide range of percepts evoked by McGurk syllables. We explored whether a deep neural network (DNN) known as AVHuBERT could provide an alternative modeling framework. AVHuBERT model variants were presented with McGurk syllables consisting of auditory “ba” paired with visual “ga” recorded from eight different talkers. AVHuBERT identified McGurk syllables as something other than “ba” at a rate of 59%, demonstrating a robust McGurk effect. The rate of the McGurk effect was similar to that observed in humans: 100 participants presented with the same McGurk syllables reported non-“ba” percepts on 56% of trials. AVHuBERT variants and humans produced a wide variety of responses to McGurk syllables, including the canonical McGurk fusion percept of “da,” responses without any initial consonant such as “ah” and responses with other initial consonants such as “fa.” The ability to predict percepts experienced by humans but not predicted by current Bayesian models suggest that DNNs and Bayesian models may provide complementary windows into the perceptual mechanisms underlying human audiovisual speech perception.

The online version contains supplementary material available at 10.3758/s13423-025-02846-8.