Mitigating Spurious Correlations in Weakly Supervised Semantic Segmentation via Cross-architecture Consistency Regularization

TL;DR

This paper introduces a novel weakly supervised semantic segmentation framework that uses cross-architecture consistency between CNNs and ViTs to reduce spurious correlations and improve segmentation quality without external priors.

Contribution

The proposed method employs a teacher-student framework with CNNs and ViTs, introducing a knowledge transfer loss for cross-architecture consistency to mitigate inherent model bias.

Findings

Improved segmentation accuracy over baseline methods.

Reduced spurious correlations in weakly supervised segmentation.

Enhanced pseudo mask quality through post-processing techniques.

Abstract

Scarcity of pixel-level labels is a significant challenge in practical scenarios. In specific domains like industrial smoke, acquiring such detailed annotations is particularly difficult and often requires expert knowledge. To alleviate this, weakly supervised semantic segmentation (WSSS) has emerged as a promising approach. However, due to the supervision gap and inherent bias in models trained with only image level labels, existing WSSS methods suffer from limitations such as incomplete foreground coverage, inaccurate object boundaries, and spurious correlations, especially in our domain, where emissions are always spatially coupled with chimneys. Previous solutions typically rely on additional priors or external knowledge to mitigate these issues, but they often lack scalability and fail to address the model's inherent bias toward co-occurring context. To address this, we propose a novel WSSS framework that directly targets the co-occurrence problem without relying on external supervision. Unlike prior methods that adopt a single network, we employ a teacher-student framework that combines CNNs and ViTs. We introduce a knowledge transfer loss that enforces cross-architecture consistency by aligning internal representations. Additionally, we incorporate post-processing techniques to address partial coverage and further improve pseudo mask quality.