Iterative multi-path tracking for video and volume segmentation with sparse point supervision

TL;DR

This paper introduces a semi-supervised framework for pixel-wise segmentation in videos and volumes using minimal 2D point supervision, leveraging iterative tracking and graph optimization to achieve state-of-the-art results.

Contribution

It presents a novel iterative multi-path tracking method that uses sparse 2D point annotations to produce accurate segmentations in video and volumetric data.

Findings

Achieves state-of-the-art segmentation accuracy with minimal supervision.

Effectively refines segmentation through iterative tracking and optimization.

Enables training of supervised classifiers using minimal annotation data.

Abstract

Recent machine learning strategies for segmentation tasks have shown great ability when trained on large pixel-wise annotated image datasets. It remains a major challenge however to aggregate such datasets, as the time and monetary cost associated with collecting extensive annotations is extremely high. This is particularly the case for generating precise pixel-wise annotations in video and volumetric image data. To this end, this work presents a novel framework to produce pixel-wise segmentations using minimal supervision. Our method relies on 2D point supervision, whereby a single 2D location within an object of interest is provided on each image of the data. Our method then estimates the object appearance in a semi-supervised fashion by learning object-image-specific features and by using these in a semi-supervised learning framework. Our object model is then used in a graph-based optimization problem that takes into account all provided locations and the image data in order to infer the complete pixel-wise segmentation. In practice, we solve this optimally as a tracking problem using a K-shortest path approach. Both the object model and segmentation are then refined iteratively to further improve the final segmentation. We show that by collecting 2D locations using a gaze tracker, our approach can provide state-of-the-art segmentations on a range of objects and image modalities (video and 3D volumes), and that these can then be used to train supervised machine learning classifiers.