Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies

TL;DR

This study compares Apple’s privacy labels with the actual privacy policies of nearly half a million apps, revealing significant discrepancies and suggesting improvements for more accurate privacy labeling.

Contribution

We developed a BERT-based model to automatically extract privacy policy features and identify inconsistencies with Apple’s privacy labels at scale.

Findings

Many apps report less data collection than policies indicate

Most apps with 'Data Not Collected' labels have conflicting policies

Discrepancies often stem from template use and unclear definitions

Abstract

Apple introduced privacy labels in Dec. 2020 as a way for developers to report the privacy behaviors of their apps. While Apple does not validate labels, they also require developers to provide a privacy policy, which offers an important comparison point. In this paper, we fine-tuned BERT-based language models to extract privacy policy features for 474,669 apps on the iOS App Store, comparing the output to the privacy labels. We identify discrepancies between the policies and the labels, particularly as they relate to data collected linked to users. We find that 228K apps' privacy policies may indicate data collection linked to users than what is reported in the privacy labels. More alarming, a large number (97%) of the apps with a Data Not Collected privacy label have a privacy policy indicating otherwise. We provide insights into potential sources for discrepancies, including the use of templates and confusion around Apple's definitions and requirements. These results suggest that significant work is still needed to help developers more accurately label their apps. Our system can be incorporated as a first-order check to inform developers when privacy labels are possibly misapplied.