Training-Free Neural Matte Extraction for Visual Effects

TL;DR

This paper presents a training-free neural matte extraction method for visual effects, leveraging deep image prior and trimap constraints to produce high-quality, temporally consistent mattes without the need for training data.

Contribution

It introduces a novel training-free approach using deep image prior and trimap interpolation, tailored for visual effects production, eliminating the need for ground truth training data.

Findings

Produces high-quality mattes without training data

Ensures temporal consistency in video processing

Simple yet effective algorithm

Abstract

Alpha matting is widely used in video conferencing as well as in movies, television, and social media sites. Deep learning approaches to the matte extraction problem are well suited to video conferencing due to the consistent subject matter (front-facing humans), however training-based approaches are somewhat pointless for entertainment videos where varied subjects (spaceships, monsters, etc.) may appear only a few times in a single movie -- if a method of creating ground truth for training exists, just use that method to produce the desired mattes. We introduce a training-free high quality neural matte extraction approach that specifically targets the assumptions of visual effects production. Our approach is based on the deep image prior, which optimizes a deep neural network to fit a single image, thereby providing a deep encoding of the particular image. We make use of the representations in the penultimate layer to interpolate coarse and incomplete "trimap" constraints. Videos processed with this approach are temporally consistent. The algorithm is both very simple and surprisingly effective.