A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents

TL;DR

This paper introduces MESc, a hierarchical deep learning framework for legal judgment prediction and explanation extraction from long, unstructured legal documents, demonstrating improved accuracy and interpretability across multiple datasets.

Contribution

The paper proposes MESc, a novel multi-stage encoder-based classification framework, and ORSE, an explanation extraction algorithm, addressing challenges of long, unstructured legal texts with minimal structural annotations.

Findings

MESc outperforms previous methods by approximately 2 points in accuracy.

ORSE improves explainability scores by an average of 50%.

The framework is effective across datasets from India, EU, and US.

Abstract

Automatic legal judgment prediction and its explanation suffer from the problem of long case documents exceeding tens of thousands of words, in general, and having a non-uniform structure. Predicting judgments from such documents and extracting their explanation becomes a challenging task, more so on documents with no structural annotation. We define this problem as "scarce annotated legal documents" and explore their lack of structural information and their long lengths with a deep-learning-based classification framework which we call MESc; "Multi-stage Encoder-based Supervised with-clustering"; for judgment prediction. We explore the adaptability of LLMs with multi-billion parameters (GPT-Neo, and GPT-J) to legal texts and their intra-domain(legal) transfer learning capacity. Alongside this, we compare their performance and adaptability with MESc and the impact of combining embeddings from their last layers. For such hierarchical models, we also propose an explanation extraction algorithm named ORSE; Occlusion sensitivity-based Relevant Sentence Extractor; based on the input-occlusion sensitivity of the model, to explain the predictions with the most relevant sentences from the document. We explore these methods and test their effectiveness with extensive experiments and ablation studies on legal documents from India, the European Union, and the United States with the ILDC dataset and a subset of the LexGLUE dataset. MESc achieves a minimum total performance gain of approximately 2 points over previous state-of-the-art proposed methods, while ORSE applied on MESc achieves a total average gain of 50% over the baseline explainability scores.