Weakly Supervised Segmentation as Semantic-Based Regularization

TL;DR

This paper introduces a neurosymbolic approach that combines fuzzy logic with foundation models like SAM to improve weakly supervised semantic segmentation, achieving state-of-the-art results.

Contribution

It proposes a novel framework that integrates logical constraints into foundation models to enhance pseudo-label quality in weak supervision.

Findings

Logic-guided fine-tuning improves pseudo-label quality.

Achieves state-of-the-art segmentation accuracy on Pascal VOC 2012.

Outperforms densely supervised baselines in experiments.

Abstract

Weakly supervised semantic segmentation (WSSS) trains dense pixel-level segmentation models from partial or coarse annotations such as bounding boxes, scribbles, or image-level tags. While recent work leverages foundation models such as the Segment Anything Model (SAM) to generate pseudo-labels, these approaches typically depend on heuristic prompt choices and offer limited ways to incorporate prior knowledge or heterogeneous labels. We address this gap by taking a neurosymbolic perspective: integrating differentiable fuzzy logic with deep segmentation models. Weak annotations and domain-specific priors are unified as continuous logical constraints that fine-tune SAM under weak supervision. The refined foundation model then produces improved pseudo-labels, from which we train a second-stage prompt-free segmentation model. Experiments on Pascal VOC 2012 and the REFUGE2 optic disc/cup segmentation dataset show that our logic-guided fine-tuning yields higher-quality pseudo-labels, leading to state-of-the-art segmentation accuracy that often exceeds densely supervised baselines.