Exploiting LLMs' Reasoning Capability to Infer Implicit Concepts in Legal Information Retrieval

TL;DR

This paper presents a legal information retrieval system that leverages large language models' reasoning abilities to infer implicit legal concepts, significantly improving retrieval accuracy over existing methods.

Contribution

The work introduces a novel retrieval approach that combines LLM-based reasoning with term expansion and query reformulation, outperforming previous models in legal datasets.

Findings

Enhanced retrieval accuracy using LLM-derived knowledge.

Outperformed all competitors in COLIEE 2022 and 2023.

Effective handling of real-life scenario queries.

Abstract

Statutory law retrieval is a typical problem in legal language processing, that has various practical applications in law engineering. Modern deep learning-based retrieval methods have achieved significant results for this problem. However, retrieval systems relying on semantic and lexical correlations often exhibit limitations, particularly when handling queries that involve real-life scenarios, or use the vocabulary that is not specific to the legal domain. In this work, we focus on overcoming this weaknesses by utilizing the logical reasoning capabilities of large language models (LLMs) to identify relevant legal terms and facts related to the situation mentioned in the query. The proposed retrieval system integrates additional information from the term--based expansion and query reformulation to improve the retrieval accuracy. The experiments on COLIEE 2022 and COLIEE 2023 datasets show that extra knowledge from LLMs helps to improve the retrieval result of both lexical and semantic ranking models. The final ensemble retrieval system outperformed the highest results among all participating teams in the COLIEE 2022 and 2023 competitions.