How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?

TL;DR

This study evaluates the effectiveness of pre-trained abstractive models and large language models like ChatGPT for legal case judgment summarization, revealing they are not yet fully reliable for automatic use due to inconsistencies.

Contribution

It provides an empirical assessment of domain-specific and general LLMs for legal summarization, highlighting current limitations and the need for human oversight.

Findings

Abstractive models outperform extractive ones on standard metrics.

Generated summaries often contain inconsistencies and hallucinations.

Models are not yet suitable for fully automatic legal summarization.

Abstract

Automatic summarization of legal case judgements has traditionally been attempted by using extractive summarization methods. However, in recent years, abstractive summarization models are gaining popularity since they can generate more natural and coherent summaries. Legal domain-specific pre-trained abstractive summarization models are now available. Moreover, general-domain pre-trained Large Language Models (LLMs), such as ChatGPT, are known to generate high-quality text and have the capacity for text summarization. Hence it is natural to ask if these models are ready for off-the-shelf application to automatically generate abstractive summaries for case judgements. To explore this question, we apply several state-of-the-art domain-specific abstractive summarization models and general-domain LLMs on Indian court case judgements, and check the quality of the generated summaries. In addition to standard metrics for summary quality, we check for inconsistencies and hallucinations in the summaries. We see that abstractive summarization models generally achieve slightly higher scores than extractive models in terms of standard summary evaluation metrics such as ROUGE and BLEU. However, we often find inconsistent or hallucinated information in the generated abstractive summaries. Overall, our investigation indicates that the pre-trained abstractive summarization models and LLMs are not yet ready for fully automatic deployment for case judgement summarization; rather a human-in-the-loop approach including manual checks for inconsistencies is more suitable at present.