Toward Robust Legal Text Formalization into Defeasible Deontic Logic using LLMs

TL;DR

This paper proposes a structured pipeline leveraging large language models to automate the formalization of legal texts into Defeasible Deontic Logic, improving accuracy, coherence, and coverage for legal informatics applications.

Contribution

It introduces a novel two-stage pipeline with a refinement step and a new success metric for better formalization of legal texts into DDL using LLMs.

Findings

LLMs can produce legal formalizations closely aligned with expert representations

The pipeline improves logical consistency and coverage of formalized legal norms

Evaluation across multiple models shows effectiveness of the approach

Abstract

We present a comprehensive approach to the automated formalization of legal texts using large language models (LLMs), targeting their transformation into Defeasible Deontic Logic (DDL). Our method employs a structured pipeline that segments complex normative language into atomic snippets, extracts deontic rules, and evaluates them for syntactic and semantic coherence. We introduce a refined success metric that more precisely captures the completeness of formalizations, and a novel two-stage pipeline with a dedicated refinement step to improve logical consistency and coverage. The evaluation procedure has been strengthened with stricter error assessment, and we provide comparative results across multiple LLM configurations, including newly released models and various prompting and fine-tuning strategies. Experiments on legal norms from the Australian Telecommunications Consumer Protections Code demonstrate that, when guided effectively, LLMs can produce formalizations that align closely with expert-crafted representations, underscoring their potential for scalable legal informatics.