Explainable Statute Prediction via Attention-based Model and LLM Prompting

TL;DR

This paper introduces two explainable models for legal statute prediction from case descriptions, combining attention mechanisms and large language model prompting to improve prediction accuracy and generate human-understandable explanations.

Contribution

It presents a novel attention-based supervised model and a zero-shot LLM prompting approach for legal statute prediction with explanations, evaluated on two datasets.

Findings

Both models outperform baselines in prediction accuracy.

Generated explanations are validated through automated and human assessments.

LLM prompting with Chain-of-Thought enhances explanation quality.

Abstract

In this paper, we explore the problem of automatic statute prediction where for a given case description, a subset of relevant statutes are to be predicted. Here, the term "statute" refers to a section, a sub-section, or an article of any specific Act. Addressing this problem would be useful in several applications such as AI-assistant for lawyers and legal question answering system. For better user acceptance of such Legal AI systems, we believe the predictions should also be accompanied by human understandable explanations. We propose two techniques for addressing this problem of statute prediction with explanations -- (i) AoS (Attention-over-Sentences) which uses attention over sentences in a case description to predict statutes relevant for it and (ii) LLMPrompt which prompts an LLM to predict as well as explain relevance of a certain statute. AoS uses smaller language models, specifically sentence transformers and is trained in a supervised manner whereas LLMPrompt uses larger language models in a zero-shot manner and explores both standard as well as Chain-of-Thought (CoT) prompting techniques. Both these models produce explanations for their predictions in human understandable forms. We compare statute prediction performance of both the proposed techniques with each other as well as with a set of competent baselines, across two popular datasets. Also, we evaluate the quality of the generated explanations through an automated counter-factual manner as well as through human evaluation.