Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese

TL;DR

This paper introduces the Audio Turing Test (ATT), a multi-dimensional evaluation framework for Chinese TTS systems that reduces subjectivity and improves robustness by focusing on human-likeness judgments, supported by a new dataset and automatic evaluation method.

Contribution

It presents the ATT framework, a new multi-dimensional Chinese corpus, and an automatic evaluation method, Auto-ATT, to better assess human-likeness of TTS systems compared to traditional MOS scores.

Findings

ATT effectively differentiates TTS models across multiple dimensions.

Auto-ATT aligns well with human judgments, enabling rapid evaluation.

The dataset and methods improve robustness and interpretability of TTS evaluation.

Abstract

Recent advances in large language models (LLMs) have significantly improved text-to-speech (TTS) systems, enhancing control over speech style, naturalness, and emotional expression, which brings TTS Systems closer to human-level performance. Although the Mean Opinion Score (MOS) remains the standard for TTS System evaluation, it suffers from subjectivity, environmental inconsistencies, and limited interpretability. Existing evaluation datasets also lack a multi-dimensional design, often neglecting factors such as speaking styles, context diversity, and trap utterances, which is particularly evident in Chinese TTS evaluation. To address these challenges, we introduce the Audio Turing Test (ATT), a multi-dimensional Chinese corpus dataset ATT-Corpus paired with a simple, Turing-Test-inspired evaluation protocol. Instead of relying on complex MOS scales or direct model comparisons, ATT asks evaluators to judge whether a voice sounds human. This simplification reduces rating bias and improves evaluation robustness. To further support rapid model development, we also finetune Qwen2-Audio-Instruct with human judgment data as Auto-ATT for automatic evaluation. Experimental results show that ATT effectively differentiates models across specific capability dimensions using its multi-dimensional design. Auto-ATT also demonstrates strong alignment with human evaluations, confirming its value as a fast and reliable assessment tool. The white-box ATT-Corpus and Auto-ATT can be found in ATT Hugging Face Collection (https://huggingface.co/collections/meituan/audio-turing-test-682446320368164faeaf38a4).