Compliance Checking with NLI: Privacy Policies vs. Regulations

TL;DR

This paper explores using Natural Language Inference models to automatically check privacy policies against regulations, comparing two training datasets and finding better real-world performance with the MNLI-trained model.

Contribution

It introduces an NLI-based approach for automated privacy policy compliance checking and evaluates the effectiveness of models trained on different datasets.

Findings

MNLI-trained model generalizes better to real-world policies.

SNLI-trained model has higher accuracy on test data.

NLI techniques can assist in legal compliance verification.

Abstract

A privacy policy is a document that states how a company intends to handle and manage their customers' personal data. One of the problems that arises with these privacy policies is that their content might violate data privacy regulations. Because of the enormous number of privacy policies that exist, the only realistic way to check for legal inconsistencies in all of them is through an automated method. In this work, we use Natural Language Inference (NLI) techniques to compare privacy regulations against sections of privacy policies from a selection of large companies. Our NLI model uses pre-trained embeddings, along with BiLSTM in its attention mechanism. We tried two versions of our model: one that was trained on the Stanford Natural Language Inference (SNLI) and the second on the Multi-Genre Natural Language Inference (MNLI) dataset. We found that our test accuracy was higher on our model trained on the SNLI, but when actually doing NLI tasks on real world privacy policies, the model trained on MNLI generalized and performed much better.