What Are the Facts? Automated Extraction of Court-Established Facts from Criminal-Court Opinions

TL;DR

This study explores automated methods to extract descriptions of criminal behaviors from Slovak court verdicts, significantly improving accuracy over simple regex approaches by employing advanced regex and large language models, with near-human performance.

Contribution

It demonstrates the feasibility of extracting detailed criminal behavior descriptions from court decisions using advanced regex and LLMs, outperforming baseline methods and approaching human annotation accuracy.

Findings

Advanced regex achieved 97% accuracy in extraction.

LLMs achieved 98.75% accuracy, matching human annotations in 91.75% of cases.

Combining regex and LLMs reached 92% accuracy, greatly surpassing baseline performance.

Abstract

Criminal justice administrative data contain only a limited amount of information about the committed offense. However, there is an unused source of extensive information in continental European courts' decisions: descriptions of criminal behaviors in verdicts by which offenders are found guilty. In this paper, we study the feasibility of extracting these descriptions from publicly available court decisions from Slovakia. We use two different approaches for retrieval: regular expressions and large language models (LLMs). Our baseline was a simple method employing regular expressions to identify typical words occurring before and after the description. The advanced regular expression approach further focused on "sparing" and its normalization (insertion of spaces between individual letters), typical for delineating the description. The LLM approach involved prompting the Gemini Flash 2.0 model to extract the descriptions using predefined instructions. Although the baseline identified descriptions in only 40.5% of verdicts, both methods significantly outperformed it, achieving 97% with advanced regular expressions and 98.75% with LLMs, and 99.5% when combined. Evaluation by law students showed that both advanced methods matched human annotations in about 90% of cases, compared to just 34.5% for the baseline. LLMs fully matched human-labeled descriptions in 91.75% of instances, and a combination of advanced regular expressions with LLMs reached 92%.