Automating Governing Knowledge Commons and Contextual Integrity (GKC-CI) Privacy Policy Annotations with Large Language Models

TL;DR

This paper shows that large language models can automatically annotate privacy policies with high accuracy, matching expert performance, thereby enabling scalable privacy analysis.

Contribution

It introduces a fine-tuned LLM approach for GKC-CI annotation, achieving expert-level accuracy and scaling privacy policy analysis.

Findings

Best model accuracy of 90.65%

Automatic annotation matches expert performance

Applied to 456 privacy policies for analysis

Abstract

Identifying contextual integrity (CI) and governing knowledge commons (GKC) parameters in privacy policy texts can facilitate normative privacy analysis. However, GKC-CI annotation has heretofore required manual or crowdsourced effort. This paper demonstrates that high-accuracy GKC-CI parameter annotation of privacy policies can be performed automatically using large language models. We fine-tune 50 open-source and proprietary models on 21,588 ground truth GKC-CI annotations from 16 privacy policies. Our best performing model has an accuracy of 90.65%, which is comparable to the accuracy of experts on the same task. We apply our best performing model to 456 privacy policies from a variety of online services, demonstrating the effectiveness of scaling GKC-CI annotation for privacy policy exploration and analysis. We publicly release our model training code, training and testing data, an annotation visualizer, and all annotated policies for future GKC-CI research.