Reducing the Annotation Effort for Video Object Segmentation Datasets

TL;DR

This paper proposes a semi-automatic annotation method for video object segmentation datasets using deep learning to generate pseudo-labels from bounding boxes, significantly reducing manual effort while maintaining high accuracy.

Contribution

It introduces a workflow that combines minimal manual annotations with deep networks to create large, challenging VOS datasets, exemplified by the new TAO-VOS benchmark.

Findings

Pseudo-labels from minimal manual masks are nearly as effective as full masks for training.

The TAO-VOS dataset is more challenging and exposes current method limitations.

Manual annotation of a single frame per object suffices for high-quality pseudo-labels.

Abstract

For further progress in video object segmentation (VOS), larger, more diverse, and more challenging datasets will be necessary. However, densely labeling every frame with pixel masks does not scale to large datasets. We use a deep convolutional network to automatically create pseudo-labels on a pixel level from much cheaper bounding box annotations and investigate how far such pseudo-labels can carry us for training state-of-the-art VOS approaches. A very encouraging result of our study is that adding a manually annotated mask in only a single video frame for each object is sufficient to generate pseudo-labels which can be used to train a VOS method to reach almost the same performance level as when training with fully segmented videos. We use this workflow to create pixel pseudo-labels for the training set of the challenging tracking dataset TAO, and we manually annotate a subset of the validation set. Together, we obtain the new TAO-VOS benchmark, which we make publicly available at www.vision.rwth-aachen.de/page/taovos. While the performance of state-of-the-art methods on existing datasets starts to saturate, TAO-VOS remains very challenging for current algorithms and reveals their shortcomings.