ForMaT: Dataset for Visually-Grounded Multilingual PDF Translation

TL;DR

ForMaT is a multilingual PDF translation dataset that preserves layout and visual features, serving as a benchmark for layout-aware translation models that integrate visual and textual context.

Contribution

The paper introduces ForMaT, a novel dataset of nearly 4,000 PDFs across 15 languages, emphasizing layout preservation and structural diversity for multimodal translation.

Findings

Current MT systems struggle with spatial grounding in PDFs.

ForMaT enables development of layout-aware translation models.

The dataset captures complex visual elements like tables and formulas.

Abstract

We present ForMaT (Format-Preserving Multilingual Translation), a parallel corpus of 3,956 PDFs across 15 language pairs that preserves original layout metadata proposed for multimodal machine translation. To ensure structural diversity in the dataset, we employ K-Medoids sampling over 45 geometric features, capturing complex elements like nested tables and formulas to focus only on visually diverse PDF documents. Our evaluation reveals that current MT systems struggle with spatial grounding and geometric synchronization, often losing the link between text and its visual context. ForMaT provides a benchmark for developing layout-aware translation models that integrate visual and textual context for high-fidelity document reconstruction.