SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment

TL;DR

SynerMedGen introduces a unified medical model that aligns understanding and generation tasks, achieving strong zero-shot performance and outperforming specialized models in medical image synthesis.

Contribution

It proposes generation-aligned understanding and a two-stage training strategy, enhancing medical image synthesis and generalization in a unified framework.

Findings

Achieves strong zero-shot performance across 22 medical image synthesis tasks.

Outperforms state-of-the-art specialized and unified medical models.

Demonstrates robust generalization to unseen datasets.

Abstract

Unifying multimodal understanding and generation is a compelling frontier that is beginning to emerge in the medical field. However, the limited existing unified medical models typically treat understanding and generation as disjoint objectives, lacking a meaningful functional synergy. In this work, we identify and address a critical question in unified medical modeling: what form of understanding truly benefits generation. We present SynerMedGen, a unified framework built on the proposed principle of generation-aligned understanding, which synergizes understanding objectives with generation tasks via task alignment. SynerMedGen introduces three generation-aligned understanding tasks and a two-stage training strategy that transfers generation-beneficial representations learned during understanding training to medical image synthesis. Remarkably, even with understanding training alone, our SynerMedGen achieves strong zero-shot performance across 22 medical image synthesis tasks and demonstrates robust generalization to unseen datasets. When combined with generation training, SynerMedGen consistently outperforms state-of-the-art specialized medical image synthesis models as well as recent unified medical models. We also release a large-scale dataset named SynerMed consisting of 1M paired synthesis samples and 2M generation-derived understanding instances to support further research on understanding-generation synergy. Our project can be accessed at https://github.com/Mhilab/SynerMedGen.