LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles

TL;DR

This paper introduces LaMP-Cap, a multimodal dataset for personalized figure caption generation that leverages figure images and profiles to produce captions closer to author style, outperforming text-only approaches.

Contribution

LaMP-Cap is the first dataset to incorporate multimodal figure profiles for personalized caption generation, enhancing the relevance and style match of generated captions.

Findings

Profile information improves caption quality.

Images in profiles are more helpful than text.

Multimodal profiles outperform text-only profiles.

Abstract

Figure captions are crucial for helping readers understand and remember a figure's key message. Many models have been developed to generate these captions, helping authors compose better quality captions more easily. Yet, authors almost always need to revise generic AI-generated captions to match their writing style and the domain's style, highlighting the need for personalization. Despite language models' personalization (LaMP) advances, these technologies often focus on text-only settings and rarely address scenarios where both inputs and profiles are multimodal. This paper introduces LaMP-Cap, a dataset for personalized figure caption generation with multimodal figure profiles. For each target figure, LaMP-Cap provides not only the needed inputs, such as figure images, but also up to three other figures from the same document--each with its image, caption, and figure-mentioning paragraphs--as a profile to characterize the context. Experiments with four LLMs show that using profile information consistently helps generate captions closer to the original author-written ones. Ablation studies reveal that images in the profile are more helpful than figure-mentioning paragraphs, highlighting the advantage of using multimodal profiles over text-only ones.