PubMed-Ophtha: An open resource for training ophthalmology vision-language models on scientific literature

TL;DR

PubMed-Ophtha is a comprehensive, high-resolution dataset of ophthalmological images and captions extracted from open-access articles, designed to facilitate training vision-language models in ophthalmology.

Contribution

The paper introduces a novel hierarchical dataset with detailed annotations and a pipeline for extracting and annotating ophthalmology figures from scientific literature.

Findings

High-quality image-caption pairs with 102,023 entries.

Achieved a mean sentence BLEU score of 0.913 for caption segmentation.

Panel and image detection models reached mAP@0.50 of 0.909 and 0.892.

Abstract

Vision-language models hold considerable promise for ophthalmology, but their development depends on large-scale, high-quality image-text datasets that remain scarce. We present PubMed-Ophtha, a hierarchical dataset of 102,023 ophthalmological image-caption pairs extracted from 15,842 open-access articles in PubMed Central. Unlike existing datasets, figures are extracted directly from article PDFs at full resolution and decomposed into their constituent panels, panel identifiers, and individual images. Each image is annotated with its imaging modality -- color fundus photography, optical coherence tomography, retinal imaging, or other -- and a mark status indicating the presence of annotation marks such as arrows. Figure captions are split into panel-level subcaptions using a two-step LLM approach, achieving a mean average sentence BLEU score of 0.913 on human-annotated data. Panel and image detection models reach a mAP@0.50 of 0.909 and 0.892, respectively, and figure extraction achieves a median IoU of 0.997. To support reproducibility, we additionally release the human-annotated ground-truth data, all trained models, and the full dataset generation pipeline.