Collecting Prosody in the Wild: A Content-Controlled, Privacy-First Smartphone Protocol and Empirical Evaluation

TL;DR

This paper presents a privacy-preserving smartphone protocol for collecting prosodic speech data using scripted sentences, enabling large-scale, standardized, and ethically compliant analysis of natural speech variations.

Contribution

It introduces a novel content-controlled, privacy-first protocol with on-device feature extraction and demonstrates its effectiveness through a large-scale empirical study.

Findings

High participant compliance and data quality

Successful prediction of speaker sex from prosodic features

Effective concurrent prediction of affective states

Abstract

Collecting everyday speech data for prosodic analysis is challenging due to the confounding of prosody and semantics, privacy constraints, and participant compliance. We introduce and empirically evaluate a content-controlled, privacy-first smartphone protocol that uses scripted read-aloud sentences to standardize lexical content (including prompt valence) while capturing natural variation in prosodic delivery. The protocol performs on-device prosodic feature extraction, deletes raw audio immediately, and transmits only derived features for analysis. We deployed the protocol in a large study (N = 560; 9,877 recordings), evaluated compliance and data quality, and conducted diagnostic prediction tasks on the extracted features, predicting speaker sex and concurrently reported momentary affective states (valence, arousal). We discuss implications and directions for advancing and deploying the protocol.