Policy Compliance Detection via Expression Tree Inference

TL;DR

This paper introduces a method to automatically infer valid expression trees from policy texts for compliance detection, improving explainability and applicability over previous approaches that relied on expert-provided trees.

Contribution

We develop a constrained decoding approach to automatically generate valid expression trees from policies, enabling scalable and explainable policy compliance detection.

Findings

63% of generated trees are logically equivalent to gold trees

88% of trees are correct according to human evaluation

Our method improves the scalability of policy compliance detection

Abstract

Policy Compliance Detection (PCD) is a task we encounter when reasoning over texts, e.g. legal frameworks. Previous work to address PCD relies heavily on modeling the task as a special case of Recognizing Textual Entailment. Entailment is applicable to the problem of PCD, however viewing the policy as a single proposition, as opposed to multiple interlinked propositions, yields poor performance and lacks explainability. To address this challenge, more recent proposals for PCD have argued for decomposing policies into expression trees consisting of questions connected with logic operators. Question answering is used to obtain answers to these questions with respect to a scenario. Finally, the expression tree is evaluated in order to arrive at an overall solution. However, this work assumes expression trees are provided by experts, thus limiting its applicability to new policies. In this work, we learn how to infer expression trees automatically from policy texts. We ensure the validity of the inferred trees by introducing constrained decoding using a finite state automaton to ensure the generation of valid trees. We determine through automatic evaluation that 63% of the expression trees generated by our constrained generation model are logically equivalent to gold trees. Human evaluation shows that 88% of trees generated by our model are correct.