# Real-Time Global Illumination Decomposition of Videos

## TL;DR

This paper introduces a real-time method for decomposing monocular videos into direct and indirect illumination layers, enabling consistent appearance editing and surpassing previous techniques in quality and speed.

## Contribution

We present the first real-time, temporally coherent decomposition of videos into illumination components using sparsity priors and a novel optimization strategy.

## Key findings

- Achieves real-time decomposition at video frame rates.
- Outperforms existing methods in quality and speed.
- Enables real-time appearance editing applications.

## Abstract

We propose the first approach for the decomposition of a monocular color video into direct and indirect illumination components in real time. We retrieve, in separate layers, the contribution made to the scene appearance by the scene reflectance, the light sources and the reflections from various coherent scene regions to one another. Existing techniques that invert global light transport require image capture under multiplexed controlled lighting, or only enable the decomposition of a single image at slow off-line frame rates. In contrast, our approach works for regular videos and produces temporally coherent decomposition layers at real-time frame rates. At the core of our approach are several sparsity priors that enable the estimation of the per-pixel direct and indirect illumination layers based on a small set of jointly estimated base reflectance colors. The resulting variational decomposition problem uses a new formulation based on sparse and dense sets of non-linear equations that we solve efficiently using a novel alternating data-parallel optimization strategy. We evaluate our approach qualitatively and quantitatively, and show improvements over the state of the art in this field, in both quality and runtime. In addition, we demonstrate various real-time appearance editing applications for videos with consistent illumination.