CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering

TL;DR

This paper introduces CGIntrinsics, a large-scale physically-based rendered dataset and a new training method that significantly improves intrinsic image decomposition, outperforming existing methods even when trained solely on synthetic data.

Contribution

The paper presents a novel synthetic dataset and an end-to-end training approach that enhances intrinsic image decomposition performance on real-world images.

Findings

Synthetic data alone outperforms previous state-of-the-art methods.

Adding real-world sparse annotations further improves results.

Carefully-rendered synthetic data is highly effective for training.

Abstract

Intrinsic image decomposition is a challenging, long-standing computer vision problem for which ground truth data is very difficult to acquire. We explore the use of synthetic data for training CNN-based intrinsic image decomposition models, then applying these learned models to real-world images. To that end, we present \ICG, a new, large-scale dataset of physically-based rendered images of scenes with full ground truth decompositions. The rendering process we use is carefully designed to yield high-quality, realistic images, which we find to be crucial for this problem domain. We also propose a new end-to-end training method that learns better decompositions by leveraging \ICG, and optionally IIW and SAW, two recent datasets of sparse annotations on real-world images. Surprisingly, we find that a decomposition network trained solely on our synthetic data outperforms the state-of-the-art on both IIW and SAW, and performance improves even further when IIW and SAW data is added during training. Our work demonstrates the suprising effectiveness of carefully-rendered synthetic data for the intrinsic images task.