STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliability

TL;DR

This paper introduces STED, a new similarity metric, and a consistency scoring framework to evaluate and improve the reliability of structured outputs generated by large language models, with practical applications in model selection and prompt tuning.

Contribution

The paper presents a novel similarity metric, STED, and a comprehensive framework for assessing and enhancing LLM output consistency, supported by extensive experiments and practical use cases.

Findings

STED outperforms existing metrics in semantic and structural similarity evaluation.

Significant variation in model consistency observed across different LLMs.

Framework enables targeted model selection and prompt refinement for reliable structured data generation.

Abstract

Large Language Models (LLMs) are increasingly deployed for structured data generation, yet output consistency remains critical for production applications. We introduce a comprehensive framework for evaluating and improving consistency in LLM-generated structured outputs. Our approach combines: (1) STED (Semantic Tree Edit Distance), a novel similarity metric balancing semantic flexibility with structural strictness when comparing JSON outputs, and (2) a consistency scoring framework aggregating multiple STED measurements across repeated generations to quantify reliability. Through systematic experiments on synthetic datasets with controlled schema, expression, and semantic variations, we demonstrate STED achieves superior performance ($0.86-0.90$ similarity for semantic equivalents, $0.0$ for structural breaks) compared to existing metrics including TED, BERTScore, and DeepDiff. Applying our framework to benchmark six LLMs reveals significant variations: Claude-3.7-Sonnet demonstrates exceptional consistency, maintaining near-perfect structural reliability even at high temperatures ($T=0.9$), while models like Claude-3-Haiku and Nova-Pro exhibit substantial degradation requiring careful tuning. Our framework enables practical applications including targeted model selection for structured tasks, iterative prompt refinement for reproducible results, and diagnostic analysis to identify inconsistency root causes. This work provides theoretical foundations and practical tools for ensuring reliable structured output generation in LLM-based production systems.