Pose2RGBD. Generating Depth and RGB images from absolute positions

TL;DR

This paper introduces Pose2RGBD, a neural network-based method that generates RGBD images from pose data, enabling scene reconstruction and navigation, with new datasets and an unsupervised training approach.

Contribution

It presents a novel neural rendering technique for RGBD generation from pose, along with two new datasets and an unsupervised dataset creation method.

Findings

Successfully generates RGBD images from pose data.

Creates synthetic and real-world datasets for training.

Demonstrates scene navigation using generated RGBD images.

Abstract

We propose a method at the intersection of Computer Vision and Computer Graphics fields, which automatically generates RGBD images using neural networks, based on previously seen and synchronized video, depth and pose signals. Since the models must be able to reconstruct both texture (RGB) and structure (Depth), it creates an implicit representation of the scene, as opposed to explicit ones, such as meshes or point clouds. The process can be thought of as neural rendering, where we obtain a function f : Pose -> RGBD, which we can use to navigate through the generated scene, similarly to graphics simulations. We introduce two new datasets, one based on synthetic data with full ground truth information, while the other one being recorded from a drone flight in an university campus, using only video and GPS signals. Finally, we propose a fully unsupervised method of generating datasets from videos alone, in order to train the Pose2RGBD networks. Code and datasets are available at:: https://gitlab.com/mihaicristianpirvu/pose2rgbd.