LLMs for Legal Subsumption in German Employment Contracts

TL;DR

This paper investigates the use of Large Language Models to classify clauses in German employment contracts as valid, unfair, or void, exploring different legal contexts to improve accuracy and assist legal professionals.

Contribution

It introduces an extended dataset with legal sources and guidelines, and evaluates LLMs' effectiveness in legal clause classification with new insights into context effects.

Findings

Full-text legal sources moderately improve LLM performance

Examination guidelines significantly boost recall and F1-score

LLMs still lag behind human lawyers in accuracy

Abstract

Legal work, characterized by its text-heavy and resource-intensive nature, presents unique challenges and opportunities for NLP research. While data-driven approaches have advanced the field, their lack of interpretability and trustworthiness limits their applicability in dynamic legal environments. To address these issues, we collaborated with legal experts to extend an existing dataset and explored the use of Large Language Models (LLMs) and in-context learning to evaluate the legality of clauses in German employment contracts. Our work evaluates the ability of different LLMs to classify clauses as "valid," "unfair," or "void" under three legal context variants: no legal context, full-text sources of laws and court rulings, and distilled versions of these (referred to as examination guidelines). Results show that full-text sources moderately improve performance, while examination guidelines significantly enhance recall for void clauses and weighted F1-Score, reaching 80\%. Despite these advancements, LLMs' performance when using full-text sources remains substantially below that of human lawyers. We contribute an extended dataset, including examination guidelines, referenced legal sources, and corresponding annotations, alongside our code and all log files. Our findings highlight the potential of LLMs to assist lawyers in contract legality review while also underscoring the limitations of the methods presented.