Generative Omnimatte: Learning to Decompose Video into Layers

TL;DR

This paper introduces a generative layered video decomposition framework that accurately separates objects and effects in videos without relying on static backgrounds or depth data, enabling high-quality editing and occlusion completion.

Contribution

It proposes a novel diffusion-based approach that does not assume scene stationarity or depth, allowing for effective decomposition and editing of complex, dynamic videos.

Findings

Produces clean, complete object layers with effects

Handles dynamic occlusions with convincing completions

Works on casually captured videos with complex effects

Abstract

Given a video and a set of input object masks, an omnimatte method aims to decompose the video into semantically meaningful layers containing individual objects along with their associated effects, such as shadows and reflections. Existing omnimatte methods assume a static background or accurate pose and depth estimation and produce poor decompositions when these assumptions are violated. Furthermore, due to the lack of generative prior on natural videos, existing methods cannot complete dynamic occluded regions. We present a novel generative layered video decomposition framework to address the omnimatte problem. Our method does not assume a stationary scene or require camera pose or depth information and produces clean, complete layers, including convincing completions of occluded dynamic regions. Our core idea is to train a video diffusion model to identify and remove scene effects caused by a specific object. We show that this model can be finetuned from an existing video inpainting model with a small, carefully curated dataset, and demonstrate high-quality decompositions and editing results for a wide range of casually captured videos containing soft shadows, glossy reflections, splashing water, and more.