Are manual annotations necessary for statutory interpretations retrieval?

TL;DR

This paper investigates whether manual annotations are essential for retrieving legal interpretations, exploring optimal annotation quantities, selection strategies, and the potential of large language models to automate annotation.

Contribution

It provides empirical insights into annotation requirements and evaluates the effectiveness of automated annotation using large language models in legal interpretation retrieval.

Findings

Optimal number of annotations per legal concept identified

Performance gains from selective annotation strategies

LLMs can effectively automate annotation process

Abstract

One of the elements of legal research is looking for cases where judges have extended the meaning of a legal concept by providing interpretations of what a concept means or does not mean. This allow legal professionals to use such interpretations as precedents as well as laymen to better understand the legal concept. The state-of-the-art approach for retrieving the most relevant interpretations for these concepts currently depends on the ranking of sentences and the training of language models over annotated examples. That manual annotation process can be quite expensive and need to be repeated for each such concept, which prompted recent research in trying to automate this process. In this paper, we highlight the results of various experiments conducted to determine the volume, scope and even the need for manual annotation. First of all, we check what is the optimal number of annotations per a legal concept. Second, we check if we can draw the sentences for annotation randomly or there is a gain in the performance of the model, when only the best candidates are annotated. As the last question we check what is the outcome of automating the annotation process with the help of an LLM.