Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers

TL;DR

This paper introduces a novel layout-guided diffusion transformer for controllable pathology image synthesis, leveraging a scalable annotation framework and achieving high fidelity, spatial control, and diagnostic consistency, with applications in data augmentation.

Contribution

The paper presents a new multi-agent annotation framework and a layout-aware diffusion transformer that enhances controllability and fidelity in pathology image generation.

Findings

Outperforms existing methods in fidelity and control

Improves diagnostic consistency of generated images

Enhances downstream task performance like cancer classification

Abstract

Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail. However, existing text-guided diffusion models offer only coarse global control and lack the ability to enforce fine-grained structural constraints. Progress is further limited by the absence of large datasets that pair patch-level spatial layouts with detailed diagnostic descriptions, since generating such annotations for gigapixel whole-slide images is prohibitively time-consuming for human experts. To overcome these challenges, we first develop a scalable multi-agent LVLM annotation framework that integrates image description, diagnostic step extraction, and automatic quality judgment into a coordinated pipeline, and we evaluate the reliability of the system through a human verification process. This framework enables efficient construction of fine-grained and clinically aligned supervision at scale. Building on the curated data, we propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer. Through hierarchical multimodal attention, IC-DiT maintains global semantic coherence while accurately preserving structural and morphological details. Extensive experiments on five histopathology datasets show that IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods. In addition, the generated images serve as effective data augmentation resources for downstream tasks such as cancer classification and survival analysis.