LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English

TL;DR

This paper introduces LexSumm, a comprehensive benchmark for legal summarization in English, and LexT5, a sequence-to-sequence model tailored for legal text, highlighting current challenges and opportunities for improvement.

Contribution

It creates the first diverse legal summarization benchmark and develops a legal-oriented sequence-to-sequence model, addressing gaps in existing legal NLP tools.

Findings

Zero-shot LLM summaries exhibit abstraction and faithfulness errors.

LexT5 outperforms encoder-only models in legal summarization tasks.

LexSumm provides a diverse dataset for evaluating legal summarization in multiple jurisdictions.

Abstract

In the evolving NLP landscape, benchmarks serve as yardsticks for gauging progress. However, existing Legal NLP benchmarks only focus on predictive tasks, overlooking generative tasks. This work curates LexSumm, a benchmark designed for evaluating legal summarization tasks in English. It comprises eight English legal summarization datasets, from diverse jurisdictions, such as the US, UK, EU and India. Additionally, we release LexT5, legal oriented sequence-to-sequence model, addressing the limitation of the existing BERT-style encoder-only models in the legal domain. We assess its capabilities through zero-shot probing on LegalLAMA and fine-tuning on LexSumm. Our analysis reveals abstraction and faithfulness errors even in summaries generated by zero-shot LLMs, indicating opportunities for further improvements. LexSumm benchmark and LexT5 model are available at https://github.com/TUMLegalTech/LexSumm-LexT5.