Cross-Policy Compliance Detection via Question Answering

TL;DR

This paper introduces a question answering approach for policy compliance detection, improving accuracy especially in cross-policy scenarios, by decomposing policies into questions and leveraging pre-trained QA models.

Contribution

It proposes a novel question answering decomposition method for policy compliance detection, enhancing accuracy and interpretability over traditional textual entailment approaches.

Findings

Improved accuracy in cross-policy compliance detection.

Effective use of pre-trained question answering models.

Enhanced interpretability and annotation efficiency.

Abstract

Policy compliance detection is the task of ensuring that a scenario conforms to a policy (e.g. a claim is valid according to government rules or a post in an online platform conforms to community guidelines). This task has been previously instantiated as a form of textual entailment, which results in poor accuracy due to the complexity of the policies. In this paper we propose to address policy compliance detection via decomposing it into question answering, where questions check whether the conditions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label. Despite the initial upfront annotation cost, we demonstrate that this approach results in better accuracy, especially in the cross-policy setup where the policies during testing are unseen in training. In addition, it allows us to use existing question answering models pre-trained on existing large datasets. Finally, it explicitly identifies the information missing from a scenario in case policy compliance cannot be determined. We conduct our experiments using a recent dataset consisting of government policies, which we augment with expert annotations and find that the cost of annotating question answering decomposition is largely offset by improved inter-annotator agreement and speed.