On the Suitability of LLM-Driven Agents for Dark Pattern Audits

TL;DR

This paper evaluates the effectiveness of LLM-driven agents in autonomously identifying and classifying dark patterns in website interfaces, specifically within the context of data rights request workflows, highlighting both capabilities and limitations.

Contribution

It introduces a novel LLM-based agent capable of navigating, evidence gathering, and classifying dark patterns in complex web workflows, demonstrating its potential and challenges.

Findings

The agent can reliably locate and complete rights request flows.

Dark pattern classification by the agent is generally consistent and reproducible.

The study identifies specific conditions where the agent's judgments are unreliable.

Abstract

As LLM-driven agents begin to autonomously navigate the web, their ability to interpret and respond to manipulative interface design becomes critical. A fundamental question that emerges is: can such agents reliably recognize patterns of friction, misdirection, and coercion in interface design (i.e., dark patterns)? We study this question in a setting where the workflows are consequential: website portals associated with the submission of CCPA-related data rights requests. These portals operationalize statutory rights, but they are implemented as interactive interfaces whose design can be structured to facilitate, burden, or subtly discourage the exercise of those rights. We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows, structured evidence gathering, and classification of potential dark patterns. Across a set of 456 data broker websites, we evaluate: (1) the ability of the agent to consistently locate and complete request flows, (2) the reliability and reproducibility of its dark pattern classifications, and (3) the conditions under which it fails or produces poor judgments. Our findings characterize both the feasibility and the limitations of using LLM-driven agents for scalable dark pattern auditing.