Structure from Collision

TL;DR

This paper introduces a novel method called SfC-NeRF for estimating both visible and invisible internal structures of objects from collision-induced appearance changes, advancing 3D reconstruction capabilities beyond surface estimation.

Contribution

The paper proposes SfC-NeRF, a new model that reconstructs internal object structures from collision videos, incorporating volume annealing to avoid local optima and handle ill-posed problems.

Findings

Successfully estimated internal structures of diverse objects

Demonstrated effectiveness on 115 objects with various properties

Showed improvement over existing 3D reconstruction methods

Abstract

Recent advancements in neural 3D representations, such as neural radiance fields (NeRF) and 3D Gaussian splatting (3DGS), have enabled the accurate estimation of 3D structures from multiview images. However, this capability is limited to estimating the visible external structure, and identifying the invisible internal structure hidden behind the surface is difficult. To overcome this limitation, we address a new task called Structure from Collision (SfC), which aims to estimate the structure (including the invisible internal structure) of an object from appearance changes during collision. To solve this problem, we propose a novel model called SfC-NeRF that optimizes the invisible internal structure of an object through a video sequence under physical, appearance (i.e., visible external structure)-preserving, and keyframe constraints. In particular, to avoid falling into undesirable local optima owing to its ill-posed nature, we propose volume annealing; that is, searching for global optima by repeatedly reducing and expanding the volume. Extensive experiments on 115 objects involving diverse structures (i.e., various cavity shapes, locations, and sizes) and material properties revealed the properties of SfC and demonstrated the effectiveness of the proposed SfC-NeRF.