RoSI: Recovering 3D Shape Interiors from Few Articulation Images

TL;DR

This paper introduces RoSI, a learning framework that reconstructs the interior structures of 3D models from exterior images in various articulated poses, enhancing the completeness of models used in gaming, VR/AR, and deep learning.

Contribution

RoSI is a novel, category-agnostic neural approach that infers interior planes from few multi-view, multi-articulation images, including motion extrapolation capabilities.

Findings

Accurately recovers interior structures from limited views.

Generalizes well to unseen object categories.

Effectively predicts part articulations and motions.

Abstract

The dominant majority of 3D models that appear in gaming, VR/AR, and those we use to train geometric deep learning algorithms are incomplete, since they are modeled as surface meshes and missing their interior structures. We present a learning framework to recover the shape interiors (RoSI) of existing 3D models with only their exteriors from multi-view and multi-articulation images. Given a set of RGB images that capture a target 3D object in different articulated poses, possibly from only few views, our method infers the interior planes that are observable in the input images. Our neural architecture is trained in a category-agnostic manner and it consists of a motion-aware multi-view analysis phase including pose, depth, and motion estimations, followed by interior plane detection in images and 3D space, and finally multi-view plane fusion. In addition, our method also predicts part articulations and is able to realize and even extrapolate the captured motions on the target 3D object. We evaluate our method by quantitative and qualitative comparisons to baselines and alternative solutions, as well as testing on untrained object categories and real image inputs to assess its generalization capabilities.