Five Years of SciCap: What We Learned and Future Directions for Scientific Figure Captioning

TL;DR

Over five years, SciCap evolved from a small project into a leading effort in scientific figure captioning, involving data collection, evaluations, challenges, and interactive tools, while identifying future research directions.

Contribution

This paper reviews five years of SciCap's development, highlighting key lessons learned, challenges faced, and proposing future directions for scientific figure captioning research.

Findings

Curated and updated a large dataset of figure-caption pairs.

Conducted extensive automatic and human evaluations.

Launched annual challenges and developed interactive captioning tools.

Abstract

Between 2021 and 2025, the SciCap project grew from a small seed-funded idea at The Pennsylvania State University (Penn State) into one of the central efforts shaping the scientific figure-captioning landscape. Supported by a Penn State seed grant, Adobe, and the Alfred P. Sloan Foundation, what began as our attempt to test whether domain-specific training, which was successful in text models like SciBERT, could also work for figure captions expanded into a multi-institution collaboration. Over these five years, we curated, released, and continually updated a large collection of figure-caption pairs from arXiv papers, conducted extensive automatic and human evaluations on both generated and author-written captions, navigated the rapid rise of large language models (LLMs), launched annual challenges, and built interactive systems that help scientists write better captions. In this piece, we look back at the first five years of SciCap and summarize the key technical and methodological lessons we learned. We then outline five major unsolved challenges and propose directions for the next phase of research in scientific figure captioning.