Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences

TL;DR

This paper introduces new models and a large open-source dataset for converting spoken mathematical expressions into LaTeX, improving accuracy and enabling multilingual and sentence-level recognition.

Contribution

It provides the first large-scale, multilingual dataset and benchmarks for speech-to-LaTeX conversion, along with novel models demonstrating significant performance improvements.

Findings

Models outperform previous approaches on the S2L-equations benchmark.

Achieved 40% CER on mathematical sentence recognition.

Demonstrated comparable results with audio language models on MathSpeech.

Abstract

Conversion of spoken mathematical expressions is a challenging task that involves transcribing speech into a strictly structured symbolic representation while addressing the ambiguity inherent in the pronunciation of equations. Although significant progress has been achieved in automatic speech recognition (ASR) and language models (LM), the problem of converting spoken mathematics into LaTeX remains underexplored. This task directly applies to educational and research domains, such as lecture transcription or note creation. Based on ASR post-correction, prior work requires 2 transcriptions, focuses only on isolated equations, has a limited test set, and provides neither training data nor multilingual coverage. To address these issues, we present the first fully open-source large-scale dataset, comprising over 66,000 human-annotated audio samples of mathematical equations and sentences in English and Russian, drawn from diverse scientific domains. In addition to the ASR post-correction models and few-shot prompting, we apply audio language models, demonstrating comparable character error rate (CER) results on the MathSpeech benchmark (28% vs. 30%) for the equations conversion. In contrast, on the proposed S2L-equations benchmark, our models outperform the MathSpeech model by a substantial margin of more than 36 percentage points, even after accounting for LaTeX formatting artifacts (27% vs. 64%). We establish the first benchmark for mathematical sentence recognition (S2L-sentences) and achieve an equation CER of 40%. This work lays the groundwork for future advances in multimodal AI, with a particular focus on mathematical content recognition.