ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contracts

TL;DR

This paper introduces ContractEval, a benchmark for evaluating open-source versus proprietary large language models in clause-level legal risk identification within commercial contracts, revealing performance gaps and tradeoffs.

Contribution

It is the first comprehensive benchmark assessing open-source LLMs in legal risk analysis, providing insights into their strengths, limitations, and areas for improvement.

Findings

Proprietary models outperform open-source models in correctness and effectiveness.

Larger models tend to perform better, but with diminishing returns.

Reasoning mode improves effectiveness but reduces correctness.

Abstract

The potential of large language models (LLMs) in specialized domains such as legal risk analysis remains underexplored. In response to growing interest in locally deploying open-source LLMs for legal tasks while preserving data confidentiality, this paper introduces ContractEval, the first benchmark to thoroughly evaluate whether open-source LLMs could match proprietary LLMs in identifying clause-level legal risks in commercial contracts. Using the Contract Understanding Atticus Dataset (CUAD), we assess 4 proprietary and 15 open-source LLMs. Our results highlight five key findings: (1) Proprietary models outperform open-source models in both correctness and output effectiveness, though some open-source models are competitive in certain specific dimensions. (2) Larger open-source models generally perform better, though the improvement slows down as models get bigger. (3) Reasoning ("thinking") mode improves output effectiveness but reduces correctness, likely due to over-complicating simpler tasks. (4) Open-source models generate "no related clause" responses more frequently even when relevant clauses are present. This suggests "laziness" in thinking or low confidence in extracting relevant content. (5) Model quantization speeds up inference but at the cost of performance drop, showing the tradeoff between efficiency and accuracy. These findings suggest that while most LLMs perform at a level comparable to junior legal assistants, open-source models require targeted fine-tuning to ensure correctness and effectiveness in high-stakes legal settings. ContractEval offers a solid benchmark to guide future development of legal-domain LLMs.