Entry Separation using a Mixed Visual and Textual Language Model: Application to 19th century French Trade Directories

TL;DR

This paper introduces a novel method combining visual cues and textual analysis using a mixed language model to accurately segment entries in 19th century French trade directories, improving data extraction from historical documents.

Contribution

It presents a new pragmatic approach that integrates visual tokens into a language model for fine-grained entry separation, leveraging both visual and textual information.

Findings

Effective on 19th century French trade directories

Outperforms traditional layout analysis methods

Available code and models facilitate replication

Abstract

When extracting structured data from repetitively organized documents, such as dictionaries, directories, or even newspapers, a key challenge is to correctly segment what constitutes the basic text regions for the target database. Traditionally, such a problem was tackled as part of the layout analysis and was mostly based on visual clues for dividing (top-down) approaches. Some agglomerating (bottom-up) approaches started to consider textual information to link similar contents, but they required a proper over-segmentation of fine-grained units. In this work, we propose a new pragmatic approach whose efficiency is demonstrated on 19th century French Trade Directories. We propose to consider two sub-problems: coarse layout detection (text columns and reading order), which is assumed to be effective and not detailed here, and a fine-grained entry separation stage for which we propose to adapt a state-of-the-art Named Entity Recognition (NER) approach. By injecting special visual tokens, coding, for instance, indentation or breaks, into the token stream of the language model used for NER purpose, we can leverage both textual and visual knowledge simultaneously. Code, data, results and models are available at https://github.com/soduco/paper-entryseg-icdar23-code, https://huggingface.co/HueyNemud/ (icdar23-entrydetector* variants)