Can you hear me $\textit{now}$? Sensitive comparisons of human and machine perception

TL;DR

This paper investigates how to better compare human and machine perception by introducing sensitive tests that reveal understanding beyond explicit reports, using adversarial speech as a case study.

Contribution

It demonstrates that humans can understand adversarial speech in various ways despite reports of unintelligibility, proposing improved methods for perception comparison.

Findings

Humans can discriminate adversarial speech from non-speech.

Humans can complete phrases begun in adversarial speech.

Humans can solve math problems in adversarial speech.

Abstract

The rise of machine-learning systems that process sensory input has brought with it a rise in comparisons between human and machine perception. But such comparisons face a challenge: Whereas machine perception of some stimulus can often be probed through direct and explicit measures, much of human perceptual knowledge is latent, incomplete, or unavailable for explicit report. Here, we explore how this asymmetry can cause such comparisons to misestimate the overlap in human and machine perception. As a case study, we consider human perception of \textit{adversarial speech} -- synthetic audio commands that are recognized as valid messages by automated speech-recognition systems but that human listeners reportedly hear as meaningless noise. In five experiments, we adapt task designs from the human psychophysics literature to show that even when subjects cannot freely transcribe such speech commands (the previous benchmark for human understanding), they often can demonstrate other forms of understanding, including discriminating adversarial speech from closely matched non-speech (Experiments 1--2), finishing common phrases begun in adversarial speech (Experiments 3--4), and solving simple math problems posed in adversarial speech (Experiment 5) -- even for stimuli previously described as unintelligible to human listeners. We recommend the adoption of such "sensitive tests" when comparing human and machine perception, and we discuss the broader consequences of such approaches for assessing the overlap between systems.