Perspective Plane Program Induction from a Single Image

TL;DR

This paper introduces P3I, a neuro-symbolic approach that infers holistic scene representations from a single image, enabling improved perspective correction and image manipulation.

Contribution

The paper proposes a novel framework, P3I, combining search and gradient methods for joint scene and camera pose inference from a single image.

Findings

P3I outperforms baselines in camera pose estimation

P3I improves global scene structure inference

P3I enhances downstream image manipulation tasks

Abstract

We study the inverse graphics problem of inferring a holistic representation for natural images. Given an input image, our goal is to induce a neuro-symbolic, program-like representation that jointly models camera poses, object locations, and global scene structures. Such high-level, holistic scene representations further facilitate low-level image manipulation tasks such as inpainting. We formulate this problem as jointly finding the camera pose and scene structure that best describe the input image. The benefits of such joint inference are two-fold: scene regularity serves as a new cue for perspective correction, and in turn, correct perspective correction leads to a simplified scene structure, similar to how the correct shape leads to the most regular texture in shape from texture. Our proposed framework, Perspective Plane Program Induction (P3I), combines search-based and gradient-based algorithms to efficiently solve the problem. P3I outperforms a set of baselines on a collection of Internet images, across tasks including camera pose estimation, global structure inference, and down-stream image manipulation tasks.