GeMM-GAN: A Multimodal Generative Model Conditioned on Histopathology Images and Clinical Descriptions for Gene Expression Profile Generation

TL;DR

GeMM-GAN is a novel multimodal generative model that synthesizes realistic gene expression profiles conditioned on histopathology images and clinical data, aiding biomedical research while respecting privacy and cost constraints.

Contribution

The paper introduces GeMM-GAN, a new GAN architecture combining image and text modalities for gene expression synthesis, outperforming existing models in realism and functional relevance.

Findings

Outperforms standard generative models by over 11% in disease prediction accuracy.

Generates more biologically coherent gene expression profiles.

Effective integration of histopathology images and clinical metadata.

Abstract

Biomedical research increasingly relies on integrating diverse data modalities, including gene expression profiles, medical images, and clinical metadata. While medical images and clinical metadata are routinely collected in clinical practice, gene expression data presents unique challenges for widespread research use, mainly due to stringent privacy regulations and costly laboratory experiments. To address these limitations, we present GeMM-GAN, a novel Generative Adversarial Network conditioned on histopathology tissue slides and clinical metadata, designed to synthesize realistic gene expression profiles. GeMM-GAN combines a Transformer Encoder for image patches with a final Cross Attention mechanism between patches and text tokens, producing a conditioning vector to guide a generative model in generating biologically coherent gene expression profiles. We evaluate our approach on the TCGA dataset and demonstrate that our framework outperforms standard generative models and generates more realistic and functionally meaningful gene expression profiles, improving by more than 11\% the accuracy on downstream disease type prediction compared to current state-of-the-art generative models. Code will be available at: https://github.com/francescapia/GeMM-GAN