LLM-based Embedders for Prior Case Retrieval

TL;DR

This paper explores the use of large language model-based text embedders for prior case retrieval in legal systems, demonstrating their effectiveness over traditional IR methods and supervised models across multiple benchmarks.

Contribution

It introduces LLM-based embedders for PCR, overcoming input length and data scarcity issues, and shows they outperform existing methods on benchmark datasets.

Findings

LLM-based embedders outperform BM25 and supervised models in PCR.

They effectively handle lengthy legal texts without truncation.

Unsupervised LLM embedders perform well across multiple datasets.

Abstract

In common law systems, legal professionals such as lawyers and judges rely on precedents to build their arguments. As the volume of cases has grown massively over time, effectively retrieving prior cases has become essential. Prior case retrieval (PCR) is an information retrieval (IR) task that aims to automatically identify the most relevant court cases for a specific query from a large pool of potential candidates. While IR methods have seen several paradigm shifts over the last few years, the vast majority of PCR methods continue to rely on traditional IR methods, such as BM25. The state-of-the-art deep learning IR methods have not been successful in PCR due to two key challenges: i. Lengthy legal text limitation; when using the powerful BERT-based transformer models, there is a limit of input text lengths, which inevitably requires to shorten the input via truncation or division with a loss of legal context information. ii. Lack of legal training data; due to data privacy concerns, available PCR datasets are often limited in size, making it difficult to train deep learning-based models effectively. In this research, we address these challenges by leveraging LLM-based text embedders in PCR. LLM-based embedders support longer input lengths, and since we use them in an unsupervised manner, they do not require training data, addressing both challenges simultaneously. In this paper, we evaluate state-of-the-art LLM-based text embedders in four PCR benchmark datasets and show that they outperform BM25 and supervised transformer-based models.