A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis

TL;DR

This paper introduces CRAFTS, a novel generative model for pathology image synthesis that ensures semantic accuracy and diversity, significantly aiding clinical AI development and diagnostic tasks.

Contribution

CRAFTS is the first pathology-specific text-to-image generative model with a novel alignment mechanism to reduce semantic drift, trained on 2.8 million image-caption pairs.

Findings

Generates diverse, high-quality pathological images across 30 cancer types.

Improves performance in classification, retrieval, and other clinical tasks.

Enables precise tissue architecture control using additional inputs.

Abstract

The development of clinical-grade artificial intelligence in pathology is limited by the scarcity of diverse, high-quality annotated datasets. Generative models offer a potential solution but suffer from semantic instability and morphological hallucinations that compromise diagnostic reliability. To address this challenge, we introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS), the first generative foundation model for pathology-specific text-to-image synthesis. By leveraging a dual-stage training strategy on approximately 2.8 million image-caption pairs, CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy. This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations. Furthermore, CRAFTS-augmented datasets enhance the performance across various clinical tasks, including classification, cross-modal retrieval, self-supervised learning, and visual question answering. In addition, coupling CRAFTS with ControlNet enables precise control over tissue architecture from inputs such as nuclear segmentation masks and fluorescence images. By overcoming the critical barriers of data scarcity and privacy concerns, CRAFTS provides a limitless source of diverse, annotated histology data, effectively unlocking the creation of robust diagnostic tools for rare and complex cancer phenotypes.