Intelligibility of Text-to-Speech Systems for Mathematical Expressions

TL;DR

This study evaluates the intelligibility of five text-to-speech models for mathematical expressions, revealing significant gaps compared to human renditions and highlighting the need for improved TTS systems for MX.

Contribution

It introduces a comprehensive evaluation framework for TTS systems on mathematical expressions, including listening tests, transcription accuracy, and comparison with expert renditions.

Findings

TTS models often produce unintelligible MX outputs.

Performance varies significantly across models and MX categories.

TTS outputs are generally worse than human expert renditions.

Abstract

There has been limited evaluation of advanced Text-to-Speech (TTS) models with Mathematical eXpressions (MX) as inputs. In this work, we design experiments to evaluate quality and intelligibility of five TTS models through listening and transcribing tests for various categories of MX. We use two Large Language Models (LLMs) to generate English pronunciation from LaTeX MX as TTS models cannot process LaTeX directly. We use Mean Opinion Score from user ratings and quantify intelligibility through transcription correctness using three metrics. We also compare listener preference of TTS outputs with respect to human expert rendition of same MX. Results establish that output of TTS models for MX is not necessarily intelligible, the gap in intelligibility varies across TTS models and MX category. For most categories, performance of TTS models is significantly worse than that of expert rendition. The effect of choice of LLM is limited. This establishes the need to improve TTS models for MX.