Deep Active Learning with Crowdsourcing Data for Privacy Policy Classification

TL;DR

This paper introduces Calpric, an active learning and crowdsourcing-based tool that efficiently classifies privacy policies with high accuracy while significantly reducing labeling effort and addressing class imbalance.

Contribution

It presents a novel combination of active learning and crowdsourcing for privacy policy classification, reducing labeling costs and improving class balance.

Findings

Achieves same F1 score with only 62% of original labels

Effectively addresses class imbalance in privacy policy data

Reduces annotation effort comparable to skilled human annotators

Abstract

Privacy policies are statements that notify users of the services' data practices. However, few users are willing to read through policy texts due to the length and complexity. While automated tools based on machine learning exist for privacy policy analysis, to achieve high classification accuracy, classifiers need to be trained on a large labeled dataset. Most existing policy corpora are labeled by skilled human annotators, requiring significant amount of labor hours and effort. In this paper, we leverage active learning and crowdsourcing techniques to develop an automated classification tool named Calpric (Crowdsourcing Active Learning PRIvacy Policy Classifier), which is able to perform annotation equivalent to those done by skilled human annotators with high accuracy while minimizing the labeling cost. Specifically, active learning allows classifiers to proactively select the most informative segments to be labeled. On average, our model is able to achieve the same F1 score using only 62% of the original labeling effort. Calpric's use of active learning also addresses naturally occurring class imbalance in unlabeled privacy policy datasets as there are many more statements stating the collection of private information than stating the absence of collection. By selecting samples from the minority class for labeling, Calpric automatically creates a more balanced training set.