Implicit Object Mapping With Noisy Data

TL;DR

This paper investigates how noise in data affects the training of Neural Radiance Fields (NeRFs) for scene understanding, proposing a pipeline to decompose scenes into individual object-NeRFs despite noisy inputs and demonstrating the importance of depth supervision and camera extrinsics.

Contribution

The authors introduce a scene decomposition pipeline that handles noisy object masks and poses, and analyze the impact of various noise sources on NeRF quality.

Findings

Depth supervision reduces sensitivity to mask noise.

Including camera extrinsics improves NeRF robustness.

The pipeline can effectively decompose scenes with noisy data.

Abstract

Modelling individual objects in a scene as Neural Radiance Fields (NeRFs) provides an alternative geometric scene representation that may benefit downstream robotics tasks such as scene understanding and object manipulation. However, we identify three challenges to using real-world training data collected by a robot to train a NeRF: (i) The camera trajectories are constrained, and full visual coverage is not guaranteed - especially when obstructions to the objects of interest are present; (ii) the poses associated with the images are noisy due to odometry or localization noise; (iii) the objects are not easily isolated from the background. This paper evaluates the extent to which above factors degrade the quality of the learnt implicit object representation. We introduce a pipeline that decomposes a scene into multiple individual object-NeRFs, using noisy object instance masks and bounding boxes, and evaluate the sensitivity of this pipeline with respect to noisy poses, instance masks, and the number of training images. We uncover that the sensitivity to noisy instance masks can be partially alleviated with depth supervision and quantify the importance of including the camera extrinsics in the NeRF optimisation process.