Helping Johnny Make Sense of Privacy Policies with LLMs

TL;DR

This paper introduces PRISMe, an LLM-based browser extension that helps users understand privacy policies through summaries and interactive exploration, supported by a user study and technical improvements.

Contribution

It presents a novel interactive tool combining LLMs with retrieval-augmented generation for privacy policy analysis, along with insights from user studies.

Findings

Users appreciated clear overviews and detailed explanations.

Identified issues with adversarial robustness and hallucinations.

Retrieval-augmented generation can improve explanation reliability.

Abstract

Understanding and engaging with privacy policies is crucial for online privacy, yet these documents remain notoriously complex and difficult to navigate. We present PRISMe, an interactive browser extension that combines LLM-based policy assessment with a dashboard and customizable chat interface, enabling users to skim quick overviews or explore policy details in depth while browsing. We conduct a user study (N=22) with participants of diverse privacy knowledge to investigate how users interpret the tool's explanations and how it shapes their engagement with privacy policies, identifying distinct interaction patterns. Participants valued the clear overviews and conversational depth, but flagged some issues, particularly adversarial robustness and hallucination risks. Thus, we investigate how a retrieval-augmented generation (RAG) approach can alleviate issues by re-running the chat queries from the study. Our findings surface design challenges as well as technical trade-offs, contributing actionable insights for developing future user-centered, trustworthy privacy policy analysis tools.