MODNet-V: Improving Portrait Video Matting via Background Restoration

TL;DR

MODNet-V enhances portrait video matting by dynamically restoring backgrounds from input videos, reducing model complexity, and incorporating a patch refinement module for high-resolution processing, all without requiring extra user input.

Contribution

The paper introduces a novel background restoration module (BRM) integrated into MODNet, creating MODNet-V, which achieves comparable or better performance with fewer parameters and supports end-to-end training.

Findings

MODNet-V has only 1/3 of MODNet's parameters.

MODNet-V achieves comparable or better performance than existing models.

The model can be trained end-to-end on a single GPU.

Abstract

To address the challenging portrait video matting problem more precisely, existing works typically apply some matting priors that require additional user efforts to obtain, such as annotated trimaps or background images. In this work, we observe that instead of asking the user to explicitly provide a background image, we may recover it from the input video itself. To this end, we first propose a novel background restoration module (BRM) to recover the background image dynamically from the input video. BRM is extremely lightweight and can be easily integrated into existing matting models. By combining BRM with a recent image matting model, MODNet, we then present MODNet-V for portrait video matting. Benefited from the strong background prior provided by BRM, MODNet-V has only 1/3 of the parameters of MODNet but achieves comparable or even better performances. Our design allows MODNet-V to be trained in an end-to-end manner on a single NVIDIA 3090 GPU. Finally, we introduce a new patch refinement module (PRM) to adapt MODNet-V for high-resolution videos while keeping MODNet-V lightweight and fast.