FigEx2: Visual-Conditioned Panel Detection and Captioning for Scientific Compound Figures

TL;DR

FigEx2 is a novel visual-conditioned framework that localizes panels and generates captions for scientific compound figures, transforming unusable images into valuable panel-text pairs for improved scientific figure understanding.

Contribution

It introduces a new detection and captioning method with a noise-aware fusion module and a curated benchmark, enabling zero-shot transfer across scientific domains.

Findings

Achieves 0.728 mAP@0.5:0.95 for panel detection.

Outperforms existing models in METEOR and BERTScore metrics.

Transfers zero-shot to physics and chemistry figures without fine-tuning.

Abstract

Scientific compound figures combine multiple labeled panels into a single image. However, in a PMC-scale crawl of 346,567 compound figures, 16.3% have no caption and 1.8% only have captions shorter than ten words, causing them to be discarded by existing caption-decomposition pipelines. We propose FigEx2, a visual-conditioned framework that localizes panels and generates panel-wise captions directly from the image, converting otherwise unusable figures into aligned panel-text pairs for downstream pretraining and retrieval. To mitigate linguistic variance in open-ended captioning, we introduce a noise-aware gated fusion module that adaptively controls how caption features condition the detection query space, and employ a staged SFT+RL strategy with CLIP-based alignment and BERTScore-based semantic rewards. To support high-quality supervision, we curate BioSci-Fig-Cap, a refined benchmark for panel-level grounding, alongside cross-disciplinary test suites in physics and chemistry. FigEx2 achieves 0.728 mAP@0.5:0.95 for detection, outperforms Qwen3-VL-8B by 0.44 in METEOR and 0.22 in BERTScore, and transfers zero-shot to out-of-distribution scientific domains without fine-tuning.