See No Evil: Semantic Context-Aware Privacy Risk Detection for AR

TL;DR

PrivAR is a novel privacy risk detection system for AR that uses vision language models with contextual reasoning to identify and obfuscate sensitive information, enhancing privacy without losing contextual cues.

Contribution

This paper introduces PrivAR, the first system leveraging chain-of-thought prompting in vision language models for context-aware privacy detection in AR environments.

Findings

PrivAR achieves 81.48% accuracy and 84.62% F1-score in privacy risk detection.

PrivAR reduces privacy leakage rate to 17.58%.

User studies show effective privacy awareness with contextually-informed warnings.

Abstract

Augmented reality (AR) systems pose unique privacy risks due to their continuous capture of visual data. Existing AR privacy frameworks lack semantic understanding of visual content, limiting their effectiveness in detecting context-dependent privacy risks. We propose PrivAR, which leverages vision language models (VLMs) with chain-of-thought prompting for contextual privacy risk detection in AR environments. PrivAR uses visual scene cues to infer potential sensitive information types, such as identifying password notes in office environments through contextual reasoning. PrivAR detects and obfuscates textual content, preventing exposure of sensitive information while preserving contextual cues necessary for VLM inference. Additionally, we investigate contextually-informed warning interfaces to enhance user privacy awareness. Experiments on a real-world AR dataset show that PrivAR achieves superior accuracy (81.48%) and F1-score (84.62%) compared to baselines, while reducing privacy leakage rate to 17.58%. User studies evaluating contextually-informed warning interfaces provide insights into effective privacy-aware AR design.