Validity Assessment of Legal Will Statements as Natural Language Inference

TL;DR

This paper introduces a new NLI dataset focused on legal will statements, highlighting the challenges neural models face in understanding complex, longer legal texts and the need for further research to improve their comprehension.

Contribution

The paper presents a novel NLI dataset for legal wills with longer texts and three-input entailment decisions, and evaluates neural models' performance and limitations on this task.

Findings

Neural models achieve over 80% macro F1 and accuracy.

Group accuracy is in the mid 80s, indicating superficial understanding.

Models sometimes rely on semantically irrelevant tokens.

Abstract

This work introduces a natural language inference (NLI) dataset that focuses on the validity of statements in legal wills. This dataset is unique because: (a) each entailment decision requires three inputs: the statement from the will, the law, and the conditions that hold at the time of the testator's death; and (b) the included texts are longer than the ones in current NLI datasets. We trained eight neural NLI models in this dataset. All the models achieve more than 80% macro F1 and accuracy, which indicates that neural approaches can handle this task reasonably well. However, group accuracy, a stricter evaluation measure that is calculated with a group of positive and negative examples generated from the same statement as a unit, is in mid 80s at best, which suggests that the models' understanding of the task remains superficial. Further ablative analyses and explanation experiments indicate that all three text segments are used for prediction, but some decisions rely on semantically irrelevant tokens. This indicates that overfitting on these longer texts likely happens, and that additional research is required for this task to be solved.