Can we make NeRF-based visual localization privacy-preserving?

TL;DR

This paper investigates privacy risks in NeRF-based visual localization and introduces a new privacy-preserving NeRF variant trained with segmentation supervision that maintains localization accuracy.

Contribution

It presents a protocol to evaluate NeRF privacy risks and proposes ppNeSF, a segmentation-based NeRF that balances privacy and localization performance.

Findings

NeRFs trained with photometric losses reveal fine scene details.

Removing color prediction heads does not prevent privacy leaks.

ppNeSF achieves state-of-the-art localization while enhancing privacy.

Abstract

Visual localization (VL) is the task of estimating the camera pose in a known scene. VL methods, a.o., can be distinguished based on how they represent the scene, e.g., explicitly through a (sparse) point cloud or a collection of images or implicitly through the weights of a neural network. Recently, NeRF-based methods have become popular for VL. While NeRFs offer high-quality novel view synthesis, they inadvertently encode fine scene details, raising privacy concerns when deployed in cloud-based localization services as sensitive information could be recovered. In this paper, we tackle this challenge on two ends. We first propose a new protocol to assess privacy-preservation of NeRF-based representations. We show that NeRFs trained with photometric losses store fine-grained details in their geometry representations, making them vulnerable to privacy attacks, even if the head that predicts colors is removed. Second, we propose ppNeSF (Privacy-Preserving Neural Segmentation Field), a NeRF variant trained with segmentation supervision instead of RGB images. These segmentation labels are learned in a self-supervised manner, ensuring they are coarse enough to obscure identifiable scene details while remaining discriminativeness in 3D. The segmentation space of ppNeSF can be used for accurate visual localization, yielding state-of-the-art results.