LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text

TL;DR

This paper introduces a new benchmark and evaluation framework for assessing the linguistic quality of legal texts generated by large language models, addressing gaps in current factual accuracy-focused metrics.

Contribution

It develops a regression-based evaluation model for legal text quality, creates a specialized legal question set, and analyzes 49 LLMs, revealing key insights about model performance and limitations.

Findings

Model quality plateaus at 14 billion parameters.

Engineering choices like quantization have minimal impact.

Reasoning models outperform base architectures.

Abstract

As large language models (LLMs) are increasingly used in legal applications, current evaluation benchmarks tend to focus mainly on factual accuracy while largely neglecting important linguistic quality aspects such as clarity, coherence, and terminology. To address this gap, we propose three steps: First, we develop a regression model to evaluate the quality of legal texts based on clarity, coherence, and terminology. Second, we create a specialized set of legal questions. Third, we analyze 49 LLMs using this evaluation framework. Our analysis identifies three key findings: First, model quality levels off at 14 billion parameters, with only a marginal improvement of $2.7\%$ noted at 72 billion parameters. Second, engineering choices such as quantization and context length have a negligible impact, as indicated by statistical significance thresholds above 0.016. Third, reasoning models consistently outperform base architectures. A significant outcome of our research is the release of a ranking list and Pareto analysis, which highlight the Qwen3 series as the optimal choice for cost-performance tradeoffs. This work not only establishes standardized evaluation protocols for legal LLMs but also uncovers fundamental limitations in current training data refinement approaches. Code and models are available at: https://github.com/lyxx3rd/LegalEval-Q.