LIGHT: Multi-Modal Text Linking on Historical Maps

TL;DR

LIGHT is a multi-modal approach that effectively links text fragments on historical maps by integrating linguistic, visual, and geometric features, outperforming existing methods in map text understanding.

Contribution

The paper introduces LIGHT, a novel multi-modal model that combines geometric, visual, and linguistic features for improved text linking on historical maps, addressing limitations of prior layout analysis methods.

Findings

LIGHT outperforms existing methods on MapText datasets.

The geometric-aware embedding improves spatial understanding of map text.

Multi-modal learning enhances text linking accuracy in complex map layouts.

Abstract

Text on historical maps provides valuable information for studies in history, economics, geography, and other related fields. Unlike structured or semi-structured documents, text on maps varies significantly in orientation, reading order, shape, and placement. Many modern methods can detect and transcribe text regions, but they struggle to effectively ``link'' the recognized text fragments, e.g., determining a multi-word place name. Existing layout analysis methods model word relationships to improve text understanding in structured documents, but they primarily rely on linguistic features and neglect geometric information, which is essential for handling map text. To address these challenges, we propose LIGHT, a novel multi-modal approach that integrates linguistic, image, and geometric features for linking text on historical maps. In particular, LIGHT includes a geometry-aware embedding module that encodes the polygonal coordinates of text regions to capture polygon shapes and their relative spatial positions on an image. LIGHT unifies this geometric information with the visual and linguistic token embeddings from LayoutLMv3, a pretrained layout analysis model. LIGHT uses the cross-modal information to predict the reading-order successor of each text instance directly with a bi-directional learning strategy that enhances sequence robustness. Experimental results show that LIGHT outperforms existing methods on the ICDAR 2024/2025 MapText Competition data, demonstrating the effectiveness of multi-modal learning for historical map text linking.