Video Decomposition Prior: A Methodology to Decompose Videos into Layers

TL;DR

This paper introduces a video decomposition prior framework that decomposes videos into layers without needing extensive external data, improving tasks like segmentation, dehazing, and relighting, and sets new benchmarks with a novel logarithmic formulation.

Contribution

The paper proposes a task-agnostic video decomposition method inspired by editing practices, eliminating the need for external datasets, and introduces a new logarithmic formulation for video relighting.

Findings

Effective decomposition into RGB layers and opacity levels.

Improved performance on video dehazing and relighting tasks.

Sets new benchmarks with the logarithmic formulation.

Abstract

In the evolving landscape of video enhancement and editing methodologies, a majority of deep learning techniques often rely on extensive datasets of observed input and ground truth sequence pairs for optimal performance. Such reliance often falters when acquiring data becomes challenging, especially in tasks like video dehazing and relighting, where replicating identical motions and camera angles in both corrupted and ground truth sequences is complicated. Moreover, these conventional methodologies perform best when the test distribution closely mirrors the training distribution. Recognizing these challenges, this paper introduces a novel video decomposition prior `VDP' framework which derives inspiration from professional video editing practices. Our methodology does not mandate task-specific external data corpus collection, instead pivots to utilizing the motion and appearance of the input video. VDP framework decomposes a video sequence into a set of multiple RGB layers and associated opacity levels. These set of layers are then manipulated individually to obtain the desired results. We addresses tasks such as video object segmentation, dehazing, and relighting. Moreover, we introduce a novel logarithmic video decomposition formulation for video relighting tasks, setting a new benchmark over the existing methodologies. We observe the property of relighting emerge as we optimize for our novel relighting decomposition formulation. We evaluate our approach on standard video datasets like DAVIS, REVIDE, & SDSD and show qualitative results on a diverse array of internet videos. Project Page - https://www.cs.umd.edu/~gauravsh/video_decomposition/index.html for video results.