# Revealing Scenes by Inverting Structure from Motion Reconstructions

## TL;DR

This paper demonstrates that 3D point clouds from structure from motion can be inverted to reconstruct scene appearance, revealing privacy risks and enabling novel view generation.

## Contribution

It introduces a novel privacy attack that reconstructs scene images from sparse SfM point clouds, handling irregular data and missing attributes.

## Key findings

- Reconstructed scene images from point clouds with high accuracy
- Identified key point attributes critical for image reconstruction
- Enabled virtual scene tours from reconstructed views

## Abstract

Many 3D vision systems localize cameras within a scene using 3D point clouds. Such point clouds are often obtained using structure from motion (SfM), after which the images are discarded to preserve privacy. In this paper, we show, for the first time, that such point clouds retain enough information to reveal scene appearance and compromise privacy. We present a privacy attack that reconstructs color images of the scene from the point cloud. Our method is based on a cascaded U-Net that takes as input, a 2D multichannel image of the points rendered from a specific viewpoint containing point depth and optionally color and SIFT descriptors and outputs a color image of the scene from that viewpoint. Unlike previous feature inversion methods, we deal with highly sparse and irregular 2D point distributions and inputs where many point attributes are missing, namely keypoint orientation and scale, the descriptor image source and the 3D point visibility. We evaluate our attack algorithm on public datasets and analyze the significance of the point cloud attributes. Finally, we show that novel views can also be generated thereby enabling compelling virtual tours of the underlying scene.