SDI-Paste: Synthetic Dynamic Instance Copy-Paste for Video Instance Segmentation

TL;DR

SDI-Paste introduces a scalable video data augmentation pipeline that synthetically generates and incorporates dynamic object instances into videos, significantly improving performance in video instance segmentation tasks.

Contribution

The paper presents a novel, scalable method for synthetic dynamic object augmentation in videos specifically designed for video instance segmentation.

Findings

Achieved +2.9 AP (6.5%) improvement on Youtube-VIS 2021.

Achieved +2.1 AP (4.9%) improvement on Youtube-VIS 2021.

Demonstrated effectiveness of synthetic dynamic augmentation in video segmentation.

Abstract

Data augmentation methods such as Copy-Paste have been studied as effective ways to expand training datasets while incurring minimal costs. While such methods have been extensively implemented for image level tasks, we found no scalable implementation of Copy-Paste built specifically for video tasks. In this paper, we leverage the recent growth in video fidelity of generative models to explore effective ways of incorporating synthetically generated objects into existing video datasets to artificially expand object instance pools. We first procure synthetic video sequences featuring objects that morph dynamically with time. Our carefully devised pipeline automatically segments then copy-pastes these dynamic instances across the frames of any target background video sequence. We name our video data augmentation pipeline Synthetic Dynamic Instance Copy-Paste, and test it on the complex task of Video Instance Segmentation which combines detection, segmentation and tracking of object instances across a video sequence. Extensive experiments on the popular Youtube-VIS 2021 dataset using two separate popular networks as baselines achieve strong gains of +2.9 AP (6.5%) and +2.1 AP (4.9%). We make our code and models publicly available.