Gamifying Video Object Segmentation

TL;DR

This paper introduces a gamified interactive approach for video object segmentation that leverages human input collected via a web game to improve segmentation accuracy while reducing annotation time.

Contribution

It presents a novel gamified method combining human visual identification with optimization techniques for more effective video object segmentation.

Findings

Better trade-off between annotation time and accuracy.

Outperforms existing interactive and automated segmentation methods.

Effective on challenging video datasets.

Abstract

Video object segmentation can be considered as one of the most challenging computer vision problems. Indeed, so far, no existing solution is able to effectively deal with the peculiarities of real-world videos, especially in cases of articulated motion and object occlusions; limitations that appear more evident when we compare their performance with the human one. However, manually segmenting objects in videos is largely impractical as it requires a lot of human time and concentration. To address this problem, in this paper we propose an interactive video object segmentation method, which exploits, on one hand, the capability of humans to identify correctly objects in visual scenes, and on the other hand, the collective human brainpower to solve challenging tasks. In particular, our method relies on a web game to collect human inputs on object locations, followed by an accurate segmentation phase achieved by optimizing an energy function encoding spatial and temporal constraints between object regions as well as human-provided input. Performance analysis carried out on challenging video datasets with some users playing the game demonstrated that our method shows a better trade-off between annotation times and segmentation accuracy than interactive video annotation and automated video object segmentation approaches.