TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs

TL;DR

TeXpert introduces a comprehensive benchmark dataset to evaluate large language models' ability to generate LaTeX code for scientific documents, revealing performance gaps and common error types across various models and complexity levels.

Contribution

The paper presents TeXpert, the first multi-level LaTeX code generation benchmark for LLMs, along with an analysis of model performance and error patterns.

Findings

LLMs perform poorly on complex LaTeX tasks compared to standard benchmarks.

Open-source models like DeepSeek v3 rival closed-source models in LaTeX generation.

Formatting and package errors are common, indicating training data limitations.

Abstract

LaTeX's precision and flexibility in typesetting have made it the gold standard for the preparation of scientific documentation. Large Language Models (LLMs) present a promising opportunity for researchers to produce publication-ready material using LaTeX with natural language instructions, yet current benchmarks completely lack evaluation of this ability. By introducing TeXpert, our benchmark dataset with natural language prompts for generating LaTeX code focused on components of scientific documents across multiple difficulty levels, we conduct an in-depth analysis of LLM performance in this regard and identify frequent error types. Our evaluation across open and closed-source LLMs highlights multiple key findings: LLMs excelling on standard benchmarks perform poorly in LaTeX generation with a significant accuracy drop-off as the complexity of tasks increases; open-source models like DeepSeek v3 and DeepSeek Coder strongly rival closed-source counterparts in LaTeX tasks; and formatting and package errors are unexpectedly prevalent, suggesting a lack of diverse LaTeX examples in the training datasets of most LLMs. Our dataset, code, and model evaluations are available at https://github.com/knowledge-verse-ai/TeXpert.