Temporally Coherent Person Matting Trained on Fake-Motion Dataset

TL;DR

This paper introduces a neural network for video person matting that achieves temporal consistency without user input, trained on synthetic data generated by a novel fake-motion algorithm.

Contribution

The authors develop a fake-motion data generation method and a neural network architecture that ensures temporal stability in video person matting without manual trimaps.

Findings

Achieves temporally stable alpha mattes in videos

Operates without user-provided trimaps

Trained effectively on synthetic fake-motion data

Abstract

We propose a novel neural-network-based method to perform matting of videos depicting people that does not require additional user input such as trimaps. Our architecture achieves temporal stability of the resulting alpha mattes by using motion-estimation-based smoothing of image-segmentation algorithm outputs, combined with convolutional-LSTM modules on U-Net skip connections. We also propose a fake-motion algorithm that generates training clips for the video-matting network given photos with ground-truth alpha mattes and background videos. We apply random motion to photos and their mattes to simulate movement one would find in real videos and composite the result with the background clips. It lets us train a deep neural network operating on videos in an absence of a large annotated video dataset and provides ground-truth training-clip foreground optical flow for use in loss functions.