Cookiescanner: An Automated Tool for Detecting and Evaluating GDPR Consent Notices on Websites

TL;DR

Cookiescanner is an automated tool that detects GDPR consent notices on websites, evaluates their features, and helps assess compliance, addressing limitations of previous manual or limited automated methods.

Contribution

We introduce cookiescanner, an automated, scalable tool for detecting and analyzing GDPR consent notices, including a new annotated dataset for benchmarking.

Findings

Filter lists have high precision but lower recall than keyword methods.

BERT achieves high precision but low recall for English notices.

Color detection of buttons is effective for identifying decline options.

Abstract

The enforcement of the GDPR led to the widespread adoption of consent notices, colloquially known as cookie banners. Studies have shown that many website operators do not comply with the law and track users prior to any interaction with the consent notice, or attempt to trick users into giving consent through dark patterns. Previous research has relied on manually curated filter lists or automated detection methods limited to a subset of websites, making research on GDPR compliance of consent notices tedious or limited. We present \emph{cookiescanner}, an automated scanning tool that detects and extracts consent notices via various methods and checks if they offer a decline option or use color diversion. We evaluated cookiescanner on a random sample of the top 10,000 websites listed by Tranco. We found that manually curated filter lists have the highest precision but recall fewer consent notices than our keyword-based methods. Our BERT model achieves high precision for English notices, which is in line with previous work, but suffers from low recall due to insufficient candidate extraction. While the automated detection of decline options proved to be challenging due to the dynamic nature of many sites, detecting instances of different colors of the buttons was successful in most cases. Besides systematically evaluating our various detection techniques, we have manually annotated 1,000 websites to provide a ground-truth baseline, which has not existed previously. Furthermore, we release our code and the annotated dataset in the interest of reproducibility and repeatability.