Mining Legal Arguments to Study Judicial Formalism

TL;DR

This paper develops automated NLP methods to analyze judicial reasoning and formalism in Czech Supreme Court decisions, creating datasets and models that improve understanding of legal argumentation and challenge existing narratives about formalism in Central and Eastern Europe.

Contribution

It introduces the MADON dataset, adapts transformer models for Czech legal texts, and presents a pipeline for classifying judicial reasoning and formalism with high accuracy.

Findings

Models detect argumentative paragraphs with 82.6% F1 score.

Legal argument classification achieves 77.5% F1 score.

Decision formalism classification reaches 83.8% F1 score.

Abstract

Courts must justify their decisions, but systematically analyzing judicial reasoning at scale remains difficult. This study tests claims about formalistic judging in Central and Eastern Europe (CEE) by developing automated methods to detect and classify judicial reasoning in decisions of Czech Supreme Courts using state-of-the-art natural language processing methods. We create the MADON dataset of 272 decisions from two Czech Supreme Courts with expert annotations of 9,183 paragraphs with eight argument types and holistic formalism labels for supervised training and evaluation. Using a corpus of 300,511 Czech court decisions, we adapt transformer LLMs to Czech legal domain through continued pretraining and we experiment with methods to address dataset imbalance including asymmetric loss and class weighting. The best models can detect argumentative paragraphs (82.6% Bal-F1), classify traditional types of legal argument (77.5% Bal-F1), and classify decisions as formalistic/non-formalistic (83.8% Bal-F1). Our three-stage pipeline combining ModernBERT, Llama 3.1, and traditional feature-based machine learning achieves promising results for decision classification while reducing computational costs and increasing explainability. Empirically, we challenge prevailing narratives about CEE formalism. We demonstrate that legal argument mining enables promising judicial philosophy classification and highlight its potential for other important tasks in computational legal studies. Our methodology can be used across jurisdictions, and our entire pipeline, datasets, guidelines, models, and source codes are available at https://github.com/trusthlt/madon.