Towards Learning a Generalizable 3D Scene Representation from 2D Observations

TL;DR

This paper presents a novel neural radiance field approach that predicts 3D workspace occupancy from 2D robot observations, generalizing across unseen scenes and outperforming traditional stereo vision in accuracy.

Contribution

It introduces a global workspace frame for occupancy prediction, enabling generalization without scene-specific fine-tuning, a significant advancement over prior camera-centric methods.

Findings

Achieves 26mm reconstruction error on real scenes

Successfully generalizes to unseen object arrangements

Validates approach on humanoid robot with real sensor data

Abstract

We introduce a Generalizable Neural Radiance Field approach for predicting 3D workspace occupancy from egocentric robot observations. Unlike prior methods operating in camera-centric coordinates, our model constructs occupancy representations in a global workspace frame, making it directly applicable to robotic manipulation. The model integrates flexible source views and generalizes to unseen object arrangements without scene-specific finetuning. We demonstrate the approach on a humanoid robot and evaluate predicted geometry against 3D sensor ground truth. Trained on 40 real scenes, our model achieves 26mm reconstruction error, including occluded regions, validating its ability to infer complete 3D occupancy beyond traditional stereo vision methods.