Privacy-preserving Reflection Rendering for Augmented Reality

TL;DR

This paper reveals privacy risks in AR reflection rendering caused by environment lighting and proposes novel defenses that effectively obfuscate sensitive scene information while maintaining visual quality.

Contribution

It introduces a new privacy attack on environment lighting in AR and develops the $IPC^{2}S$ and $R^2$ defenses to protect sensitive scene data.

Findings

Privacy attack can extract sensitive scene info from environment lighting.

Proposed defenses significantly reduce automatic info extraction success.

Visual quality of reflections is preserved with high SSIM scores.

Abstract

Many augmented reality (AR) applications rely on omnidirectional environment lighting to render photorealistic virtual objects. When the virtual objects consist of reflective materials, such as a metallic sphere, the required lighting information to render such objects can consist of privacy-sensitive information that is outside the current camera view. In this paper, we show, for the first time, that accuracy-driven multi-view environment lighting can reveal out-of-camera scene information and compromise privacy. We present a simple yet effective privacy attack that extracts sensitive scene information such as human face and text information from the rendered objects, under a number of application scenarios. To defend against such attacks, we develop a novel $IPC^{2}S$ defense and a conditional $R^2$ defense. Our $IPC^{2}S$ defense, used in conjunction with a generic lighting reconstruction method, preserves the scene geometry while obfuscating the privacy-sensitive information. As a proof-of-concept, we leverage existing OCR and face detection models to identify text and human faces from past camera observations and blur the color pixels associated with detected regions. We evaluate the visual quality impact of our defense by comparing rendered virtual objects to ones rendered with a generic multi-lighting reconstruction technique, ARKit, and $R^2$ defense. Our visual and quantitative results demonstrate that our defense leads to structurally similar reflections with up to 0.98 SSIM score across a variety of rendering scenarios while preserving sensitive information by reducing the automatic extraction success rate to at most 8.8%.