Conversational Speech Naturalness Predictor

TL;DR

This paper introduces a dual-channel naturalness predictor for two-speaker conversations that significantly improves correlation with human judgments over existing methods.

Contribution

It proposes a novel dual-channel estimator with pre-trained encoders and data augmentation to better assess conversational naturalness.

Findings

Existing predictors have low or negative correlation with human ratings.

The proposed model outperforms existing predictors in correlation with human judgments.

The model is effective across in-domain and out-of-domain data.

Abstract

Evaluation of conversational naturalness is essential for developing human-like speech agents. However, existing speech naturalness predictors are often designed to assess utterances from a single speaker, failing to capture conversation-level naturalness qualities. In this paper, we present a framework for an automatic naturalness predictor for two-speaker, multi-turn conversations. We first show that existing naturalness estimators have low, or sometimes even negative, correlations with conversational naturalness, based on conversational recordings annotated with human ratings. We then propose a dual-channel naturalness estimator, in which we investigate multiple pre-trained encoders with data augmentation. Our proposed model achieves substantially higher correlation with human judgments compared to existing naturalness predictors for both in-domain and out-of-domain conditions.