Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs

TL;DR

Fleming-VL is a comprehensive multimodal medical AI framework that unifies understanding of diverse medical data types, achieving state-of-the-art results in various benchmarks through extensive pretraining and fine-tuning strategies.

Contribution

The paper introduces Fleming-VL, a novel unified framework for medical visual reasoning across heterogeneous modalities, addressing domain gaps and data format inconsistencies.

Findings

Achieves state-of-the-art performance on medical VQA, video QA, and 3D image understanding benchmarks.

Effectively integrates long-context data from natural and medical domains for pretraining.

Demonstrates the benefits of combining supervised fine-tuning with group relative policy optimization.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable effectiveness in various general-domain scenarios, such as visual question answering and image captioning. Recently, researchers have increasingly focused on empowering MLLMs with medical conversational abilities, which hold significant promise for clinical applications. However, medical data presents unique challenges due to its heterogeneous nature -- encompassing diverse modalities including 2D images, 3D volumetric scans, and temporal video sequences. The substantial domain gap and data format inconsistencies across these modalities have hindered the development of unified medical MLLMs. To address these challenges, we propose Fleming-VL, a unified end-to-end framework for comprehensive medical visual understanding across heterogeneous modalities. Fleming-VL tackles this problem from a data-centric perspective through three key strategies: (1) scaling up pretraining by integrating long-context data from both natural and medical-specific domains; (2) complementing fine-tuning with rare medical data, including holistic video analysis and underrepresented 2D modalities such as ultrasound and dermoscopy images; (3) extending existing evaluation frameworks to incorporate 3D volumetric and video understanding benchmarks. Through supervised fine-tuning (SFT) and group relative policy optimization (GRPO), we develop Fleming-VL in multiple model scales. Extensive experiments demonstrate that Fleming-VL achieves state-of-the-art performance across multiple benchmarks, including medical VQA, video QA, and 3D medical image understanding. We publicly release Fleming-VL to promote transparent, reproducible, and auditable progress in medical AI.