LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

TL;DR

LLMStructBench is a new benchmark for evaluating large language models on extracting structured data from text and generating valid JSON, highlighting the importance of prompting strategies over model size.

Contribution

It introduces a comprehensive benchmark and dataset for assessing LLMs on structured data extraction, along with new metrics and insights into prompting strategies.

Findings

Prompting strategy selection impacts parsing reliability more than model size.

Smaller models can achieve structural validity with proper prompts.

Trade-offs exist between validity and semantic accuracy.

Abstract

We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.