Authoring image decompositions with generative models

TL;DR

This paper introduces a method for authoring layered image decompositions using generative models, enabling flexible multi-layer representations without requiring physical interpretability.

Contribution

It develops a novel convolutional VAE architecture and a framework for creating layered image decompositions guided by proxy examples.

Findings

The conv-VAE reconstructs high-fidelity images effectively.

The method can generate layered decompositions that explain input images.

It extends intrinsic image decomposition to multiple layers with generative models.

Abstract

We show how to extend traditional intrinsic image decompositions to incorporate further layers above albedo and shading. It is hard to obtain data to learn a multi-layer decomposition. Instead, we can learn to decompose an image into layers that are "like this" by authoring generative models for each layer using proxy examples that capture the Platonic ideal (Mondrian images for albedo; rendered 3D primitives for shading; material swatches for shading detail). Our method then generates image layers, one from each model, that explain the image. Our approach rests on innovation in generative models for images. We introduce a Convolutional Variational Auto Encoder (conv-VAE), a novel VAE architecture that can reconstruct high fidelity images. The approach is general, and does not require that layers admit a physical interpretation.