Operationalizing Contextual Integrity in Privacy-Conscious Assistants

TL;DR

This paper introduces a method to align AI assistants' information sharing with privacy norms using the framework of contextual integrity, evaluated through a novel webform benchmark and prompting strategies.

Contribution

It operationalizes the concept of contextual integrity to steer AI assistants' information sharing, enhancing privacy compliance in complex tasks.

Findings

Prompting LLMs with CI-based reasoning improves privacy compliance.

The novel webform benchmark effectively evaluates privacy-aware assistant behavior.

Strategies based on CI can significantly reduce inappropriate information sharing.

Abstract

Advanced AI assistants combine frontier LLMs and tool access to autonomously perform complex tasks on behalf of users. While the helpfulness of such assistants can increase dramatically with access to user information including emails and documents, this raises privacy concerns about assistants sharing inappropriate information with third parties without user supervision. To steer information-sharing assistants to behave in accordance with privacy expectations, we propose to operationalize contextual integrity (CI), a framework that equates privacy with the appropriate flow of information in a given context. In particular, we design and evaluate a number of strategies to steer assistants' information-sharing actions to be CI compliant. Our evaluation is based on a novel form filling benchmark composed of human annotations of common webform applications, and it reveals that prompting frontier LLMs to perform CI-based reasoning yields strong results.