Investigating the Impact of Dark Patterns on LLM-Based Web Agents

TL;DR

This study investigates how dark patterns in web interfaces influence the decision-making of LLM-based web agents, revealing significant susceptibility and highlighting the need for improved defenses against deceptive UI tactics.

Contribution

First comprehensive analysis of dark pattern effects on LLM web agents using a novel framework and controlled environment, revealing vulnerabilities and influencing factors.

Findings

Agents are susceptible to dark patterns 41% of the time with a single pattern

Visual and HTML modifications can alter agent susceptibility

Multiple dark patterns increase the likelihood of manipulation

Abstract

As users increasingly turn to large language model (LLM) based web agents to automate online tasks, agents may encounter dark patterns: deceptive user interface designs that manipulate users into making unintended decisions. Although dark patterns primarily target human users, their potentially harmful impacts on LLM-based generalist web agents remain unexplored. In this paper, we present the first study that investigates the impact of dark patterns on the decision-making process of LLM-based generalist web agents. To achieve this, we introduce LiteAgent, a lightweight framework that automatically prompts agents to execute tasks while capturing comprehensive logs and screen-recordings of their interactions. We also present TrickyArena, a controlled environment comprising web applications from domains such as e-commerce, streaming services, and news platforms, each containing diverse and realistic dark patterns that can be selectively enabled or disabled. Using LiteAgent and TrickyArena, we conduct multiple experiments to assess the impact of both individual and combined dark patterns on web agent behavior. We evaluate six popular LLM-based generalist web agents across three LLMs and discover that when there is a single dark pattern present, agents are susceptible to it an average of 41% of the time. We also find that modifying dark pattern UI attributes through visual design changes or HTML code adjustments and introducing multiple dark patterns simultaneously can influence agent susceptibility. This study emphasizes the need for holistic defense mechanisms in web agents, encompassing both agent-specific protections and broader web safety measures.