Augmented Question-guided Retrieval (AQgR) of Indian Case Law with LLM, RAG, and Structured Summaries

TL;DR

This paper introduces a novel retrieval system combining LLMs, RAG, and structured summaries to improve Indian case law retrieval by focusing on legal issues and providing explanations, significantly outperforming existing benchmarks.

Contribution

It presents the Augmented Question-guided Retrieval (AQgR) framework that enhances case law retrieval with legal issue focus and explanation generation, a novel approach in legal AI.

Findings

Achieved MAP score of 0.36, surpassing the benchmark of 0.1573.

Effectively identified relevant legal issues and provided explanations.

Improved relevance and interpretability in legal case retrieval.

Abstract

Identifying relevant legal precedents remains challenging, as most retrieval methods emphasize factual similarity over legal issues, and current systems often lack explanations clarifying case relevance. This paper proposes the use of Large Language Models (LLMs) to address this gap by facilitating the retrieval of relevant cases, generating explanations to elucidate relevance, and identifying core legal issues all autonomously, without requiring legal expertise. Our approach combines Retrieval Augmented Generation (RAG) with structured summaries optimized for Indian case law. Leveraging the Augmented Question-guided Retrieval (AQgR) framework, the system generates targeted legal questions based on factual scenarios to identify relevant case law more effectively. The structured summaries were assessed manually by legal experts, given the absence of a suitable structured summary dataset. Case law retrieval was evaluated using the FIRE dataset, and explanations were reviewed by legal experts, as explanation generation alongside case retrieval is an emerging innovation. Experimental evaluation on a subset of the FIRE 2019 dataset yielded promising outcomes, achieving a Mean Average Precision (MAP) score of 0.36 and a Mean Average Recall (MAR) of 0.67 across test queries, significantly surpassing the current MAP benchmark of 0.1573. This work introduces a suite of novel contributions to advance case law retrieval. By transitioning from fact-based to legal-issue-based retrieval, the proposed approach delivers more contextually relevant results that align closely with legal professionals' needs. Integrating legal questions within the retrieval process through the AQgR framework ensures more precise and meaningful retrieval by refining the context of queries.