Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects

TL;DR

This paper presents a novel neuro-implicit method for separable 3D reconstruction of two interacting objects from multi-view images, effectively handling occlusions and enabling high-quality novel view synthesis.

Contribution

It introduces a new end-to-end trainable framework with alpha-blending regularization for separating and reconstructing two interacting objects in 3D.

Findings

Significant improvements in 3D reconstruction accuracy.

Effective separation of objects despite severe occlusions.

Successful application to both rigid and articulated objects.

Abstract

Separable 3D reconstruction of multiple objects from multi-view RGB images -- resulting in two different 3D shapes for the two objects with a clear separation between them -- remains a sparsely researched problem. It is challenging due to severe mutual occlusions and ambiguities along the objects' interaction boundaries. This paper investigates the setting and introduces a new neuro-implicit method that can reconstruct the geometry and appearance of two objects undergoing close interactions while disjoining both in 3D, avoiding surface inter-penetrations and enabling novel-view synthesis of the observed scene. The framework is end-to-end trainable and supervised using a novel alpha-blending regularisation that ensures that the two geometries are well separated even under extreme occlusions. Our reconstruction method is markerless and can be applied to rigid as well as articulated objects. We introduce a new dataset consisting of close interactions between a human and an object and also evaluate on two scenes of humans performing martial arts. The experiments confirm the effectiveness of our framework and substantial improvements using 3D and novel view synthesis metrics compared to several existing approaches applicable in our setting.