Associative3D: Volumetric Reconstruction from Sparse Views

TL;DR

Associative3D introduces a joint reasoning framework for 3D volumetric scene reconstruction from two sparse views, estimating scene structure, camera parameters, and object relationships to improve reconstruction accuracy.

Contribution

The paper presents a novel joint reasoning approach that simultaneously estimates scene reconstructions, camera and object distributions, and inter-view affinities for better 3D reconstruction from sparse views.

Findings

Successfully reconstructs indoor scenes from two views

Joint reasoning improves scene consistency

Method outperforms baseline approaches

Abstract

This paper studies the problem of 3D volumetric reconstruction from two views of a scene with an unknown camera. While seemingly easy for humans, this problem poses many challenges for computers since it requires simultaneously reconstructing objects in the two views while also figuring out their relationship. We propose a new approach that estimates reconstructions, distributions over the camera/object and camera/camera transformations, as well as an inter-view object affinity matrix. This information is then jointly reasoned over to produce the most likely explanation of the scene. We train and test our approach on a dataset of indoor scenes, and rigorously evaluate the merits of our joint reasoning approach. Our experiments show that it is able to recover reasonable scenes from sparse views, while the problem is still challenging. Project site: https://jasonqsy.github.io/Associative3D