Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization

TL;DR

Magenta Green Screen is a machine learning-based matting method that captures high-quality alpha channels and color images simultaneously using a novel lighting setup, eliminating the need for special cameras or manual keying.

Contribution

It introduces a spectrally multiplexed lighting technique combined with ML colorization to achieve real-time, high-quality alpha matting without specialized equipment.

Findings

High-quality alpha channels obtained from simple lighting setup

Convincing, temporally stable colorization results

Enhanced training data for future ML matting algorithms

Abstract

We introduce Magenta Green Screen, a novel machine learning--enabled matting technique for recording the color image of a foreground actor and a simultaneous high-quality alpha channel without requiring a special camera or manual keying techniques. We record the actor on a green background but light them with only red and blue foreground lighting. In this configuration, the green channel shows the actor silhouetted against a bright, even background, which can be used directly as a holdout matte, the inverse of the actor's alpha channel. We then restore the green channel of the foreground using a machine learning colorization technique. We train the colorization model with an example sequence of the actor lit by white lighting, yielding convincing and temporally stable colorization results. We further show that time-multiplexing the lighting between Magenta Green Screen and Green Magenta Screen allows the technique to be practiced under what appears to be mostly normal lighting. We demonstrate that our technique yields high-quality compositing results when implemented on a modern LED virtual production stage. The alpha channel data obtainable with our technique can provide significantly higher quality training data for natural image matting algorithms to support future ML matting research.