Objective Metrics to Evaluate Residual-Echo Suppression During Double-Talk

TL;DR

This paper introduces two new objective metrics, DSML and RESL, to separately evaluate speech quality and residual-echo suppression during double-talk, showing they align well with human ratings and aid system tuning.

Contribution

The paper proposes novel metrics that independently measure speech preservation and echo suppression, improving evaluation accuracy over traditional SDR and aiding system design.

Findings

DSML and RESL correlate strongly with DNSMOS scores.

Metrics generalize well across different setups.

Practical scheme for dynamic system tuning is provided.

Abstract

Human subjective evaluation is optimal to assess speech quality for human perception. The recently introduced deep noise suppression mean opinion score (DNSMOS) metric was shown to estimate human ratings with great accuracy. The signal-to-distortion ratio (SDR) metric is widely used to evaluate residual-echo suppression (RES) systems by estimating speech quality during double-talk. However, since the SDR is affected by both speech distortion and residual-echo presence, it does not correlate well with human ratings according to the DNSMOS. To address that, we introduce two objective metrics to separately quantify the desired-speech maintained level (DSML) and residual-echo suppression level (RESL) during double-talk. These metrics are evaluated using a deep learning-based RES-system with a tunable design parameter. Using 280 hours of real and simulated recordings, we show that the DSML and RESL correlate well with the DNSMOS with high generalization to various setups. Also, we empirically investigate the relation between tuning the RES-system design parameter and the DSML-RESL tradeoff it creates and offer a practical design scheme for dynamic system requirements.