How Many Views Are Needed to Reconstruct an Unknown Object Using NeRF?

TL;DR

This paper introduces a non-iterative view planning method for NeRF-based object reconstruction, predicting the number of views needed based on object complexity to improve efficiency and reduce planning time.

Contribution

The paper presents PRVNet, a neural network that predicts the required number of views for object reconstruction, enabling more efficient and tailored view planning in NeRF applications.

Findings

PRVNet accurately predicts view requirements based on object complexity.

The proposed method reduces movement cost and planning time compared to baselines.

The approach generalizes well to real-world object reconstruction scenarios.

Abstract

Neural Radiance Fields (NeRFs) are gaining significant interest for online active object reconstruction due to their exceptional memory efficiency and requirement for only posed RGB inputs. Previous NeRF-based view planning methods exhibit computational inefficiency since they rely on an iterative paradigm, consisting of (1) retraining the NeRF when new images arrive; and (2) planning a path to the next best view only. To address these limitations, we propose a non-iterative pipeline based on the Prediction of the Required number of Views (PRV). The key idea behind our approach is that the required number of views to reconstruct an object depends on its complexity. Therefore, we design a deep neural network, named PRVNet, to predict the required number of views, allowing us to tailor the data acquisition based on the object complexity and plan a globally shortest path. To train our PRVNet, we generate supervision labels using the ShapeNet dataset. Simulated experiments show that our PRV-based view planning method outperforms baselines, achieving good reconstruction quality while significantly reducing movement cost and planning time. We further justify the generalization ability of our approach in a real-world experiment.