An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps

TL;DR

This paper introduces a lightweight, adaptable system that combines fine-tuned CLIP and GPT-3.5 to automatically generate descriptive and storytelling captions for historical maps, overcoming challenges of map complexity and limited training data.

Contribution

It presents a novel decision tree architecture and a hybrid captioning approach that enhances map understanding and storytelling, with minimal fine-tuning and high adaptability.

Findings

System effectively generates relevant map captions.

Invariance to text alterations in maps demonstrated.

Open-source implementation available for extension.

Abstract

Historical maps provide valuable information and knowledge about the past. However, as they often feature non-standard projections, hand-drawn styles, and artistic elements, it is challenging for non-experts to identify and interpret them. While existing image captioning methods have achieved remarkable success on natural images, their performance on maps is suboptimal as maps are underrepresented in their pre-training process. Despite the recent advance of GPT-4 in text recognition and map captioning, it still has a limited understanding of maps, as its performance wanes when texts (e.g., titles and legends) in maps are missing or inaccurate. Besides, it is inefficient or even impractical to fine-tune the model with users' own datasets. To address these problems, we propose a novel and lightweight map-captioning counterpart. Specifically, we fine-tune the state-of-the-art vision-language model CLIP to generate captions relevant to historical maps and enrich the captions with GPT-3.5 to tell a brief story regarding where, what, when and why of a given map. We propose a novel decision tree architecture to only generate captions relevant to the specified map type. Our system shows invariance to text alterations in maps. The system can be easily adapted and extended to other map types and scaled to a larger map captioning system. The code is open-sourced at https://github.com/claudaff/automatic-map-storytelling.