WeClick: Weakly-Supervised Video Semantic Segmentation with Click Annotations

TL;DR

WeClick introduces a weakly-supervised video semantic segmentation method using click annotations, leveraging memory flow knowledge distillation to improve accuracy and achieve real-time performance with minimal annotation effort.

Contribution

The paper presents a novel weakly-supervised segmentation pipeline that uses click annotations and memory flow distillation to enhance video segmentation accuracy.

Findings

Outperforms state-of-the-art methods by 10.24% mIoU on Cityscapes and Camvid.

Achieves real-time inference with low-cost click annotations.

Effectively exploits temporal information through memory flow knowledge distillation.

Abstract

Compared with tedious per-pixel mask annotating, it is much easier to annotate data by clicks, which costs only several seconds for an image. However, applying clicks to learn video semantic segmentation model has not been explored before. In this work, we propose an effective weakly-supervised video semantic segmentation pipeline with click annotations, called WeClick, for saving laborious annotating effort by segmenting an instance of the semantic class with only a single click. Since detailed semantic information is not captured by clicks, directly training with click labels leads to poor segmentation predictions. To mitigate this problem, we design a novel memory flow knowledge distillation strategy to exploit temporal information (named memory flow) in abundant unlabeled video frames, by distilling the neighboring predictions to the target frame via estimated motion. Moreover, we adopt vanilla knowledge distillation for model compression. In this case, WeClick learns compact video semantic segmentation models with the low-cost click annotations during the training phase yet achieves real-time and accurate models during the inference period. Experimental results on Cityscapes and Camvid show that WeClick outperforms the state-of-the-art methods, increases performance by 10.24% mIoU than baseline, and achieves real-time execution.