Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Models

TL;DR

Casper is a browser extension that sanitizes user prompts by removing sensitive information before sending them to web-based LLMs, thereby enhancing user privacy without requiring changes to LLM services.

Contribution

This paper introduces Casper, a novel prompt sanitization system that operates locally on users' devices using a multi-layered approach to protect privacy in web-based LLM interactions.

Findings

Achieves 98.5% accuracy in filtering PII

Filters 89.9% of privacy-sensitive topics effectively

Operates entirely on the user's device without modifying LLM services

Abstract

Web-based Large Language Model (LLM) services have been widely adopted and have become an integral part of our Internet experience. Third-party plugins enhance the functionalities of LLM by enabling access to real-world data and services. However, the privacy consequences associated with these services and their third-party plugins are not well understood. Sensitive prompt data are stored, processed, and shared by cloud-based LLM providers and third-party plugins. In this paper, we propose Casper, a prompt sanitization technique that aims to protect user privacy by detecting and removing sensitive information from user inputs before sending them to LLM services. Casper runs entirely on the user's device as a browser extension and does not require any changes to the online LLM services. At the core of Casper is a three-layered sanitization mechanism consisting of a rule-based filter, a Machine Learning (ML)-based named entity recognizer, and a browser-based local LLM topic identifier. We evaluate Casper on a dataset of 4000 synthesized prompts and show that it can effectively filter out Personal Identifiable Information (PII) and privacy-sensitive topics with high accuracy, at 98.5% and 89.9%, respectively.