Leveraging Motion Priors in Videos for Improving Human Segmentation

TL;DR

This paper introduces a novel method that leverages motion priors from videos using optical flow and reinforcement learning to improve human segmentation, especially in weakly-supervised and cross-modal scenarios.

Contribution

It proposes a memory-network-based policy to select high-quality motion segments for finetuning, enhancing segmentation performance across diverse datasets and modalities.

Findings

Improves human segmentation accuracy in videos with motion priors.

Enhances performance in RGB and Infrared modalities.

Complementary to existing domain adaptation methods.

Abstract

Despite many advances in deep-learning based semantic segmentation, performance drop due to distribution mismatch is often encountered in the real world. Recently, a few domain adaptation and active learning approaches have been proposed to mitigate the performance drop. However, very little attention has been made toward leveraging information in videos which are naturally captured in most camera systems. In this work, we propose to leverage "motion prior" in videos for improving human segmentation in a weakly-supervised active learning setting. By extracting motion information using optical flow in videos, we can extract candidate foreground motion segments (referred to as motion prior) potentially corresponding to human segments. We propose to learn a memory-network-based policy model to select strong candidate segments (referred to as strong motion prior) through reinforcement learning. The selected segments have high precision and are directly used to finetune the model. In a newly collected surveillance camera dataset and a publicly available UrbanStreet dataset, our proposed method improves the performance of human segmentation across multiple scenes and modalities (i.e., RGB to Infrared (IR)). Last but not least, our method is empirically complementary to existing domain adaptation approaches such that additional performance gain is achieved by combining our weakly-supervised active learning approach with domain adaptation approaches.