StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs

TL;DR

StructEval is a new benchmark that systematically evaluates large language models' ability to generate and convert various structured formats, revealing significant performance gaps and challenges in producing correct visual content.

Contribution

It introduces a comprehensive benchmark with novel metrics for assessing LLMs' structural output capabilities across 18 formats and 44 task types.

Findings

State-of-the-art models score only 75.58 on average.

Generation tasks are more challenging than conversion tasks.

Producing visual content is more difficult than text-only structures.

Abstract

As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and \textbf{2)} conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps-even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.