Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes

TL;DR

This paper introduces a data-driven method for selecting optimal viewpoints in 3D scenes to enhance the quality of rendered images for vision tasks, demonstrating improved segmentation performance.

Contribution

It presents a novel approach to viewpoint set selection based on statistical content matching, specifically modeling semantic object distributions within scenes.

Findings

Selected views match the example distribution according to earth mover's distance.

The approach improves semantic segmentation performance over other view selection methods.

The method is validated on SUNCG and NYUDv2 datasets.

Abstract

The use of rendered images, whether from completely synthetic datasets or from 3D reconstructions, is increasingly prevalent in vision tasks. However, little attention has been given to how the selection of viewpoints affects the performance of rendered training sets. In this paper, we propose a data-driven approach to view set selection. Given a set of example images, we extract statistics describing their contents and generate a set of views matching the distribution of those statistics. Motivated by semantic segmentation tasks, we model the spatial distribution of each semantic object category within an image view volume. We provide a search algorithm that generates a sampling of likely candidate views according to the example distribution, and a set selection algorithm that chooses a subset of the candidates that jointly cover the example distribution. Results of experiments with these algorithms on SUNCG indicate that they are indeed able to produce view distributions similar to an example set from NYUDv2 according to the earth mover's distance. Furthermore, the selected views improve performance on semantic segmentation compared to alternative view selection algorithms.