Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval

TL;DR

This paper introduces a novel few-shot method using large language models to generate interpretable and reliable relevance judgments in legal case retrieval, mimicking human reasoning and enhancing transparency.

Contribution

It proposes a multi-stage, expert-aligned approach for legal relevance judgments using LLMs, including knowledge transfer to smaller models with minimal supervision.

Findings

LLMs can produce relevance judgments comparable to human experts.

The approach improves interpretability and transparency in legal case retrieval.

Knowledge distillation enables smaller models to acquire case analysis skills.

Abstract

Determining which legal cases are relevant to a given query involves navigating lengthy texts and applying nuanced legal reasoning. Traditionally, this task has demanded significant time and domain expertise to identify key Legal Facts and reach sound juridical conclusions. In addition, existing data with legal case similarities often lack interpretability, making it difficult to understand the rationale behind relevance judgments. With the growing capabilities of large language models (LLMs), researchers have begun investigating their potential in this domain. Nonetheless, the method of employing a general large language model for reliable relevance judgments in legal case retrieval remains largely unexplored. To address this gap in research, we propose a novel few-shot approach where LLMs assist in generating expert-aligned interpretable relevance judgments. The proposed approach decomposes the judgment process into several stages, mimicking the workflow of human annotators and allowing for the flexible incorporation of expert reasoning to improve the accuracy of relevance judgments. Importantly, it also ensures interpretable data labeling, providing transparency and clarity in the relevance assessment process. Through a comparison of relevance judgments made by LLMs and human experts, we empirically demonstrate that the proposed approach can yield reliable and valid relevance assessments. Furthermore, we demonstrate that with minimal expert supervision, our approach enables a large language model to acquire case analysis expertise and subsequently transfers this ability to a smaller model via annotation-based knowledge distillation.