Cross-Domain Image Synthesis: Generating H&E from Multiplex Biomarker Imaging

TL;DR

This paper introduces a multi-level VQGAN model for generating high-quality virtual H&E stains from multiplex immunofluorescence images, enabling better integration of molecular and morphological tissue analysis.

Contribution

The study demonstrates that a multi-level VQGAN outperforms standard cGANs in producing virtual H&E images that are more useful for downstream diagnostic tasks.

Findings

VQGAN-generated images improve nuclei segmentation accuracy.

VQGAN preserves tissue semantics better than cGAN.

Virtual stains facilitate integration of molecular and morphological data.

Abstract

While multiplex immunofluorescence (mIF) imaging provides deep, spatially-resolved molecular data, integrating this information with the morphological standard of Hematoxylin & Eosin (H&E) can be very important for obtaining complementary information about the underlying tissue. Generating a virtual H&E stain from mIF data offers a powerful solution, providing immediate morphological context. Crucially, this approach enables the application of the vast ecosystem of H&E-based computer-aided diagnosis (CAD) tools to analyze rich molecular data, bridging the gap between molecular and morphological analysis. In this work, we investigate the use of a multi-level Vector-Quantized Generative Adversarial Network (VQGAN) to create high-fidelity virtual H&E stains from mIF images. We rigorously evaluated our VQGAN against a standard conditional GAN (cGAN) baseline on two publicly available colorectal cancer datasets, assessing performance on both image similarity and functional utility for downstream analysis. Our results show that while both architectures produce visually plausible images, the virtual stains generated by our VQGAN provide a more effective substrate for computer-aided diagnosis. Specifically, downstream nuclei segmentation and semantic preservation in tissue classification tasks performed on VQGAN-generated images demonstrate superior performance and agreement with ground-truth analysis compared to those from the cGAN. This work establishes that a multi-level VQGAN is a robust and superior architecture for generating scientifically useful virtual stains, offering a viable pathway to integrate the rich molecular data of mIF into established and powerful H&E-based analytical workflows.