Multi-Plane Program Induction with 3D Box Priors

TL;DR

This paper introduces Box Program Induction (BPI), a novel method that infers structured 3D scene representations from a single image, enabling advanced editing and understanding of multi-plane scenes with a box prior.

Contribution

The work presents a new approach for modeling multiple 2D planes and their 3D poses from a single image using program induction with a box prior, extending beyond prior single-plane methods.

Findings

Successfully infers multi-plane 3D scene representations from a single image.

Enables 3D-aware editing such as inpainting and camera adjustments.

Demonstrates improved scene understanding over prior single-plane models.

Abstract

We consider two important aspects in understanding and editing images: modeling regular, program-like texture or patterns in 2D planes, and 3D posing of these planes in the scene. Unlike prior work on image-based program synthesis, which assumes the image contains a single visible 2D plane, we present Box Program Induction (BPI), which infers a program-like scene representation that simultaneously models repeated structure on multiple 2D planes, the 3D position and orientation of the planes, and camera parameters, all from a single image. Our model assumes a box prior, i.e., that the image captures either an inner view or an outer view of a box in 3D. It uses neural networks to infer visual cues such as vanishing points, wireframe lines to guide a search-based algorithm to find the program that best explains the image. Such a holistic, structured scene representation enables 3D-aware interactive image editing operations such as inpainting missing pixels, changing camera parameters, and extrapolate the image contents.