VitaminP: cross-modal learning enables whole-cell segmentation from routine histology

TL;DR

VitaminP is a novel cross-modal learning framework that enables accurate whole-cell segmentation from routine H&E histology images by leveraging paired mIF data to transfer molecular boundary information.

Contribution

It introduces a general cross-modal supervision strategy for biological structure recovery, trained on extensive datasets, outperforming existing methods and supporting broad application.

Findings

VitaminP outperforms four state-of-the-art segmentation methods.

It generalizes well to unseen datasets, including rare cancer types.

VitaminPScope provides an open-source platform for scalable inference.

Abstract

Accurate whole-cell and nuclear segmentation is essential for precision pathology and spatial omics, yet routine hematoxylin and eosin (H&E) staining provides limited cytoplasmic contrast, restricting analyses to nuclei. Multiplex immunofluorescence (mIF) facilitates precise whole-cell delineation but remains constrained by cost and accessibility. We introduce VitaminP, a cross-modal learning framework enabling whole cell segmentation from H&E images. By learning from paired H&E-mIF data, VitaminP transfers molecular boundary information from mIF to overcome cytoplasmic contrast in H&E, establishing cross-modal supervision as a general strategy for recovering missing biological structure. We train VitaminP on 14 public datasets covering 34 cancer types and over 7 million instances, integrating publicly available labels with extensive annotations generated in this study, forming one of the largest resources for segmentation. VitaminP outperforms four state-of-the-art methods and generalizes to unseen datasets, including an in-house dataset spanning 24 rare cancer types. We further developed VitaminPScope, an open-source platform providing an interface for scalable inference and enabling broad adoption.