LLMs for Law: Evaluating Legal-Specific LLMs on Contract Understanding

TL;DR

This paper evaluates 10 legal-specific language models on contract understanding tasks, demonstrating their superiority over general-purpose models and establishing new state-of-the-art results in legal NLP.

Contribution

It provides the first comprehensive comparison of legal-specific LLMs on contract tasks, highlighting their effectiveness and guiding future development.

Findings

Legal-specific LLMs outperform general-purpose models on contract tasks.

Legal-BERT and Contracts-BERT set new SOTA results.

Legal-specific models achieve high accuracy with fewer parameters.

Abstract

Despite advances in legal NLP, no comprehensive evaluation covering multiple legal-specific LLMs currently exists for contract classification tasks in contract understanding. To address this gap, we present an evaluation of 10 legal-specific LLMs on three English language contract understanding tasks and compare them with 7 general-purpose LLMs. The results show that legal-specific LLMs consistently outperform general-purpose models, especially on tasks requiring nuanced legal understanding. Legal-BERT and Contracts-BERT establish new SOTAs on two of the three tasks, despite having 69% fewer parameters than the best-performing general-purpose LLM. We also identify CaseLaw-BERT and LexLM as strong additional baselines for contract understanding. Our results provide a holistic evaluation of legal-specific LLMs and will facilitate the development of more accurate contract understanding systems.