FactorMatte: Redefining Video Matting for Re-Composition Tasks

TL;DR

FactorMatte introduces a novel video matting approach that decomposes scenes into independent components, effectively handling complex interactions like reflections and shadows, and excels in re-composition and downstream tasks without requiring large external datasets.

Contribution

It presents a new factor matting formulation and a training method that works per-video, enabling disentanglement of complex scene interactions without external data or 3D knowledge.

Findings

Outperforms existing methods in classical video matting and background subtraction.

Successfully disentangles scenes with complex interactions such as splashes and reflections.

Enhances downstream tasks through improved scene decomposition.

Abstract

We propose "factor matting", an alternative formulation of the video matting problem in terms of counterfactual video synthesis that is better suited for re-composition tasks. The goal of factor matting is to separate the contents of video into independent components, each visualizing a counterfactual version of the scene where contents of other components have been removed. We show that factor matting maps well to a more general Bayesian framing of the matting problem that accounts for complex conditional interactions between layers. Based on this observation, we present a method for solving the factor matting problem that produces useful decompositions even for video with complex cross-layer interactions like splashes, shadows, and reflections. Our method is trained per-video and requires neither pre-training on external large datasets, nor knowledge about the 3D structure of the scene. We conduct extensive experiments, and show that our method not only can disentangle scenes with complex interactions, but also outperforms top methods on existing tasks such as classical video matting and background subtraction. In addition, we demonstrate the benefits of our approach on a range of downstream tasks. Please refer to our project webpage for more details: https://factormatte.github.io