Structural Causal 3D Reconstruction

TL;DR

This paper proposes a novel approach to unsupervised 3D object reconstruction from single images by structuring the latent space to encode causal relationships, improving disentanglement and reconstruction quality.

Contribution

It introduces the idea of using a topological causal ordering in latent space as an implicit regularization for 3D reconstruction, which is a new perspective compared to explicit regularizations.

Findings

Latent space structure acts as an effective implicit regularizer.

Different causal orderings significantly impact reconstruction quality.

Task-dependent causal factor ordering improves disentanglement.

Abstract

This paper considers the problem of unsupervised 3D object reconstruction from in-the-wild single-view images. Due to ambiguity and intrinsic ill-posedness, this problem is inherently difficult to solve and therefore requires strong regularization to achieve disentanglement of different latent factors. Unlike existing works that introduce explicit regularizations into objective functions, we look into a different space for implicit regularization -- the structure of latent space. Specifically, we restrict the structure of latent space to capture a topological causal ordering of latent factors (i.e., representing causal dependency as a directed acyclic graph). We first show that different causal orderings matter for 3D reconstruction, and then explore several approaches to find a task-dependent causal factor ordering. Our experiments demonstrate that the latent space structure indeed serves as an implicit regularization and introduces an inductive bias beneficial for reconstruction.