Federated Unsupervised Semantic Segmentation

TL;DR

This paper introduces FUSS, a novel federated learning framework for unsupervised semantic segmentation that aligns features and prototypes across distributed clients without supervision, improving segmentation performance in heterogeneous data settings.

Contribution

FUSS is the first fully decentralized, label-free semantic segmentation framework that promotes global feature and centroid consistency across clients.

Findings

FUSS outperforms local training and classical FL algorithms across datasets.

FUSS maintains high segmentation accuracy under data heterogeneity.

The framework is validated on both benchmark and real-world datasets.

Abstract

This work explores the application of Federated Learning (FL) to Unsupervised Semantic image Segmentation (USS). Recent USS methods extract pixel-level features using frozen visual foundation models and refine them through self-supervised objectives that encourage semantic grouping. These features are then grouped to semantic clusters to produce segmentation masks. Extending these ideas to federated settings requires feature representation and cluster centroid alignment across distributed clients, an inherently difficult task under heterogeneous data distributions in the absence of supervision. To address this, we propose FUSS (Federated Unsupervised image Semantic Segmentation) which is, to our knowledge, the first framework to enable fully decentralized, label-free semantic segmentation training. FUSS introduces novel federation strategies that promote global consistency in feature and prototype space, jointly optimizing local segmentation heads and shared semantic centroids. Experiments on both benchmark and real-world datasets, including binary and multi-class segmentation tasks, show that FUSS consistently outperforms local-only client trainings as well as extensions of classical FL algorithms under varying client data distributions. To fully support reproducibility, the source code, data partitioning scripts, and implementation details are publicly available at: https://github.com/evanchar/FUSS