Semantics through Time: Semi-supervised Segmentation of Aerial Videos with Iterative Label Propagation

TL;DR

This paper introduces SegProp, an iterative flow-based method for automatic semantic label propagation in aerial videos, significantly reducing manual annotation effort and improving segmentation accuracy, supported by a new high-resolution dataset.

Contribution

The paper presents SegProp, a novel semi-supervised label propagation technique linked to spectral clustering, and introduces Ruralscapes, a large high-resolution aerial video dataset with dense annotations.

Findings

SegProp achieves over 90% accuracy in propagating labels to unlabeled frames.

Integrating other methods into SegProp boosts label propagation performance.

Training neural networks on SegProp-labeled data improves results on new videos.

Abstract

Semantic segmentation is a crucial task for robot navigation and safety. However, current supervised methods require a large amount of pixelwise annotations to yield accurate results. Labeling is a tedious and time consuming process that has hampered progress in low altitude UAV applications. This paper makes an important step towards automatic annotation by introducing SegProp, a novel iterative flow-based method, with a direct connection to spectral clustering in space and time, to propagate the semantic labels to frames that lack human annotations. The labels are further used in semi-supervised learning scenarios. Motivated by the lack of a large video aerial dataset, we also introduce Ruralscapes, a new dataset with high resolution (4K) images and manually-annotated dense labels every 50 frames - the largest of its kind, to the best of our knowledge. Our novel SegProp automatically annotates the remaining unlabeled 98% of frames with an accuracy exceeding 90% (F-measure), significantly outperforming other state-of-the-art label propagation methods. Moreover, when integrating other methods as modules inside SegProp's iterative label propagation loop, we achieve a significant boost over the baseline labels. Finally, we test SegProp in a full semi-supervised setting: we train several state-of-the-art deep neural networks on the SegProp-automatically-labeled training frames and test them on completely novel videos. We convincingly demonstrate, every time, a significant improvement over the supervised scenario.