Occlusion-Aware Video Object Inpainting

TL;DR

This paper introduces occlusion-aware video object inpainting that recovers complete object shape and appearance in videos, using a new benchmark and a joint shape and texture completion method to handle occlusions effectively.

Contribution

The paper presents the first large-scale benchmark for video object inpainting and proposes a novel joint shape and texture completion method for occlusion-aware inpainting.

Findings

Outperforms strong baselines on YouTube-VOI benchmark

Effectively handles complex and dynamic objects

Degrades gracefully with inaccurate masks

Abstract

Conventional video inpainting is neither object-oriented nor occlusion-aware, making it liable to obvious artifacts when large occluded object regions are inpainted. This paper presents occlusion-aware video object inpainting, which recovers both the complete shape and appearance for occluded objects in videos given their visible mask segmentation. To facilitate this new research, we construct the first large-scale video object inpainting benchmark YouTube-VOI to provide realistic occlusion scenarios with both occluded and visible object masks available. Our technical contribution VOIN jointly performs video object shape completion and occluded texture generation. In particular, the shape completion module models long-range object coherence while the flow completion module recovers accurate flow with sharp motion boundary, for propagating temporally-consistent texture to the same moving object across frames. For more realistic results, VOIN is optimized using both T-PatchGAN and a new spatio-temporal attention-based multi-class discriminator. Finally, we compare VOIN and strong baselines on YouTube-VOI. Experimental results clearly demonstrate the efficacy of our method including inpainting complex and dynamic objects. VOIN degrades gracefully with inaccurate input visible mask.