The Projection-Enhancement Network (PEN)

TL;DR

The paper introduces the Projection Enhancement Network (PEN), a convolutional module that converts sub-sampled 3D microscopy data into 2D semantic images, significantly improving segmentation performance in crowded cell environments.

Contribution

PEN is a novel module that enhances 2D segmentation of 3D microscopy data by encoding depth information, improving accuracy especially in high-density cell scenarios.

Findings

PEN improves segmentation accuracy over maximum intensity projections.

PEN encodes depth information that benefits CellPose but not Mask-RCNN.

PEN enhances segmentation in crowded cell environments.

Abstract

Contemporary approaches to instance segmentation in cell science use 2D or 3D convolutional networks depending on the experiment and data structures. However, limitations in microscopy systems or efforts to prevent phototoxicity commonly require recording sub-optimally sampled data regimes that greatly reduces the utility of such 3D data, especially in crowded environments with significant axial overlap between objects. In such regimes, 2D segmentations are both more reliable for cell morphology and easier to annotate. In this work, we propose the Projection Enhancement Network (PEN), a novel convolutional module which processes the sub-sampled 3D data and produces a 2D RGB semantic compression, and is trained in conjunction with an instance segmentation network of choice to produce 2D segmentations. Our approach combines augmentation to increase cell density using a low-density cell image dataset to train PEN, and curated datasets to evaluate PEN. We show that with PEN, the learned semantic representation in CellPose encodes depth and greatly improves segmentation performance in comparison to maximum intensity projection images as input, but does not similarly aid segmentation in region-based networks like Mask-RCNN. Finally, we dissect the segmentation strength against cell density of PEN with CellPose on disseminated cells from side-by-side spheroids. We present PEN as a data-driven solution to form compressed representations of 3D data that improve 2D segmentations from instance segmentation networks.