{\epsilon}-Seg: Sparsely Supervised Semantic Segmentation of Microscopy Data

TL;DR

{ extepsilon}-Seg introduces a hierarchical variational autoencoder-based approach for sparse supervision in semantic segmentation of complex microscopy images, achieving competitive results with minimal labeled data.

Contribution

The paper presents { extepsilon}-Seg, a novel method combining HVAE, contrastive learning, and a GMM prior for effective sparse supervision in biological image segmentation.

Findings

Achieves competitive segmentation with only 0.05% labels

Effective on electron and fluorescence microscopy data

Outperforms baseline methods in sparse label scenarios

Abstract

Semantic segmentation of electron microscopy (EM) images of biological samples remains a challenge in the life sciences. EM data captures details of biological structures, sometimes with such complexity that even human observers can find it overwhelming. We introduce {\epsilon}-Seg, a method based on hierarchical variational autoencoders (HVAEs), employing center-region masking, sparse label contrastive learning (CL), a Gaussian mixture model (GMM) prior, and clustering-free label prediction. Center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, even if training labels are sparse (0.05% of the total image data or less). For optimal performance, we employ CL and a GMM prior to shape the latent space of the HVAE such that encoded input patches tend to cluster wrt. the semantic classes we wish to distinguish. Finally, instead of clustering latent embeddings for semantic segmentation, we propose a MLP semantic segmentation head to directly predict class labels from latent embeddings. We show empirical results of {\epsilon}-Seg and baseline methods on 2 dense EM datasets of biological tissues and demonstrate the applicability of our method also on fluorescence microscopy data. Our results show that {\epsilon}-Seg is capable of achieving competitive sparsely-supervised segmentation results on complex biological image data, even if only limited amounts of training labels are available.