Intent Classification and Slot Filling for Privacy Policies

TL;DR

This paper introduces PolicyIE, a new annotated corpus for intent classification and slot filling in privacy policies, along with baseline neural models and analysis of their performance and challenges.

Contribution

The creation of the PolicyIE corpus with extensive annotations and the comparison of joint sequence tagging and Seq2Seq models for privacy policy understanding.

Findings

Seq2Seq outperforms sequence tagging in slot filling

Both models perform similarly in intent classification

The corpus presents real-world challenges with limited labeled data

Abstract

Understanding privacy policies is crucial for users as it empowers them to learn about the information that matters to them. Sentences written in a privacy policy document explain privacy practices, and the constituent text spans convey further specific information about that practice. We refer to predicting the privacy practice explained in a sentence as intent classification and identifying the text spans sharing specific information as slot filling. In this work, we propose PolicyIE, an English corpus consisting of 5,250 intent and 11,788 slot annotations spanning 31 privacy policies of websites and mobile applications. PolicyIE corpus is a challenging real-world benchmark with limited labeled examples reflecting the cost of collecting large-scale annotations from domain experts. We present two alternative neural approaches as baselines, (1) intent classification and slot filling as a joint sequence tagging and (2) modeling them as a sequence-to-sequence (Seq2Seq) learning task. The experiment results show that both approaches perform comparably in intent classification, while the Seq2Seq method outperforms the sequence tagging approach in slot filling by a large margin. We perform a detailed error analysis to reveal the challenges of the proposed corpus.