JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

TL;DR

This paper introduces JSONSchemaBench, a comprehensive benchmark for evaluating constrained decoding methods in language models, focusing on efficiency, coverage, and quality across real-world JSON schemas.

Contribution

It presents a systematic evaluation framework and a large-scale benchmark for constrained decoding, providing insights into their performance and limitations in structured generation tasks.

Findings

Guidance and Outlines frameworks show high constraint compliance.

Performance varies significantly across different schema complexities.

The benchmark reveals key limitations in current constrained decoding methods.

Abstract

Reliably generating structured outputs has become a critical capability for modern language model (LM) applications. Constrained decoding has emerged as the dominant technology across sectors for enforcing structured outputs during generation. Despite its growing adoption, little has been done with the systematic evaluation of the behaviors and performance of constrained decoding. Constrained decoding frameworks have standardized around JSON Schema as a structured data format, with most uses guaranteeing constraint compliance given a schema. However, there is poor understanding of the effectiveness of the methods in practice. We present an evaluation framework to assess constrained decoding approaches across three critical dimensions: efficiency in generating constraint-compliant outputs, coverage of diverse constraint types, and quality of the generated outputs. To facilitate this evaluation, we introduce JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity. We pair the benchmark with the existing official JSON Schema Test Suite and evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. Through extensive experiments, we gain insights into the capabilities and limitations of constrained decoding on structured generation with real-world JSON schemas. Our work provides actionable insights for improving constrained decoding frameworks and structured generation tasks, setting a new standard for evaluating constrained decoding and structured generation. We release JSONSchemaBench at https://github.com/guidance-ai/jsonschemabench