Successes and critical failures of neural networks in capturing human-like speech recognition

TL;DR

This paper evaluates how well neural networks mimic human speech recognition robustness, revealing both similarities and critical failures, and suggests new directions for improving artificial auditory systems.

Contribution

It systematically compares neural network performance to human speech perception, identifying where models succeed and fail in capturing human-like robustness.

Findings

Neural networks replicate some human perceptual phenomena.

Models show robustness at certain spectrotemporal granularities.

All models fail to recover speech perceptually where humans do.

Abstract

Natural and artificial audition can in principle acquire different solutions to a given problem. The constraints of the task, however, can nudge the cognitive science and engineering of audition to qualitatively converge, suggesting that a closer mutual examination would potentially enrich artificial hearing systems and process models of the mind and brain. Speech recognition - an area ripe for such exploration - is inherently robust in humans to a number transformations at various spectrotemporal granularities. To what extent are these robustness profiles accounted for by high-performing neural network systems? We bring together experiments in speech recognition under a single synthesis framework to evaluate state-of-the-art neural networks as stimulus-computable, optimized observers. In a series of experiments, we (1) clarify how influential speech manipulations in the literature relate to each other and to natural speech, (2) show the granularities at which machines exhibit out-of-distribution robustness, reproducing classical perceptual phenomena in humans, (3) identify the specific conditions where model predictions of human performance differ, and (4) demonstrate a crucial failure of all artificial systems to perceptually recover where humans do, suggesting alternative directions for theory and model building. These findings encourage a tighter synergy between the cognitive science and engineering of audition.