Factoring Statutory Reasoning as Language Understanding Challenges

TL;DR

This paper breaks down statutory reasoning into four language understanding challenges, enhancing model performance and interpretability by leveraging structured annotations and subtasks.

Contribution

It introduces a novel decomposition of statutory reasoning into structured subtasks with annotations, improving model diagnostics and performance.

Findings

Models benefit from structured annotations

Decomposition improves model diagnostics

Incremental progress is facilitated by subtasks

Abstract

Statutory reasoning is the task of determining whether a legal statute, stated in natural language, applies to the text description of a case. Prior work introduced a resource that approached statutory reasoning as a monolithic textual entailment problem, with neural baselines performing nearly at-chance. To address this challenge, we decompose statutory reasoning into four types of language-understanding challenge problems, through the introduction of concepts and structure found in Prolog programs. Augmenting an existing benchmark, we provide annotations for the four tasks, and baselines for three of them. Models for statutory reasoning are shown to benefit from the additional structure, improving on prior baselines. Further, the decomposition into subtasks facilitates finer-grained model diagnostics and clearer incremental progress.