ReConvNet: Video Object Segmentation with Spatio-Temporal Features Modulation

TL;DR

ReConvNet is a recurrent convolutional network that efficiently adapts to new objects in video segmentation using spatio-temporal features and conditional affine transformations, achieving competitive results without online fine-tuning.

Contribution

It introduces a self-adapting spatio-temporal feature learning method for video object segmentation that does not require extra training at inference.

Findings

Competitive on DAVIS2016 without online fine-tuning

Outperforms state-of-the-art on DAVIS2017

Achieved 10th place in DAVIS-Challenge 2018

Abstract

We introduce ReConvNet, a recurrent convolutional architecture for semi-supervised video object segmentation that is able to fast adapt its features to focus on any specific object of interest at inference time. Generalization to new objects never observed during training is known to be a hard task for supervised approaches that would need to be retrained. To tackle this problem, we propose a more efficient solution that learns spatio-temporal features self-adapting to the object of interest via conditional affine transformations. This approach is simple, can be trained end-to-end and does not necessarily require extra training steps at inference time. Our method shows competitive results on DAVIS2016 with respect to state-of-the art approaches that use online fine-tuning, and outperforms them on DAVIS2017. ReConvNet shows also promising results on the DAVIS-Challenge 2018 winning the $10$-th position.