Decoding Speaker-Normalized Pitch from EEG for Mandarin Perception

TL;DR

This study demonstrates that EEG signals can be used to decode speaker-normalized pitch contours in Mandarin, revealing neural encoding of relative pitch independent of speaker variation.

Contribution

Introduces the CE-ViViT model for decoding speaker-normalized pitch contours directly from EEG data, advancing understanding of neural pitch perception.

Findings

Decodes pitch contours with modest errors

Speaker-normalized pitch contours are decoded more accurately

Supports neural encoding of relative pitch

Abstract

The same speech content produced by different speakers exhibits significant differences in pitch contour, yet listeners' semantic perception remains unaffected. This phenomenon may stem from the brain's perception of pitch contours being independent of individual speakers' pitch ranges. In this work, we recorded electroencephalogram (EEG) while participants listened to Mandarin monosyllables with varying tones, phonemes, and speakers. The CE-ViViT model is proposed to decode raw or speaker-normalized pitch contours directly from EEG. Experimental results demonstrate that the proposed model can decode pitch contours with modest errors, achieving performance comparable to state-of-the-art EEG regression methods. Moreover, speaker-normalized pitch contours were decoded more accurately, supporting the neural encoding of relative pitch.