Boundary-Refined Prototype Generation: A General End-to-End Paradigm for Semi-Supervised Semantic Segmentation

TL;DR

This paper introduces an end-to-end boundary-refined prototype generation method for semi-supervised semantic segmentation, improving boundary delineation and robustness by online clustering and adaptive prototype optimization.

Contribution

The proposed BRPG method integrates prototype generation into training, enhances boundary accuracy through confidence-based clustering, and adapts prototype numbers for scattered features, outperforming existing methods.

Findings

Outperforms state-of-the-art on PASCAL VOC 2012, Cityscapes, MS COCO.

Demonstrates robustness across diverse datasets and networks.

Achieves significant boundary refinement in segmentation results.

Abstract

Semi-supervised semantic segmentation has attracted increasing attention in computer vision, aiming to leverage unlabeled data through latent supervision. To achieve this goal, prototype-based classification has been introduced and achieved lots of success. However, the current approaches isolate prototype generation from the main training framework, presenting a non-end-to-end workflow. Furthermore, most methods directly perform the K-Means clustering on features to generate prototypes, resulting in their proximity to category semantic centers, while overlooking the clear delineation of class boundaries. To address the above problems, we propose a novel end-to-end boundary-refined prototype generation (BRPG) method. Specifically, we perform online clustering on sampled features to incorporate the prototype generation into the whole training framework. In addition, to enhance the classification boundaries, we sample and cluster high- and low-confidence features separately based on confidence estimation, facilitating the generation of prototypes closer to the class boundaries. Moreover, an adaptive prototype optimization strategy is proposed to increase the number of prototypes for categories with scattered feature distributions, which further refines the class boundaries. Extensive experiments demonstrate the remarkable robustness and scalability of our method across diverse datasets, segmentation networks, and semi-supervised frameworks, outperforming the state-of-the-art approaches on three benchmark datasets: PASCAL VOC 2012, Cityscapes and MS COCO. The code is available at https://github.com/djh-dzxw/BRPG.