Linguistically Informed Multimodal Fusion for Vietnamese Scene-Text Image Captioning: Dataset, Graph Framework, and Phonological Attention

TL;DR

This paper introduces a linguistically informed multimodal fusion framework for Vietnamese scene-text image captioning, addressing language-specific challenges with a new dataset, graph models, and phonological reasoning.

Contribution

It proposes HSTFG and PhonoSTFG, novel graph-based fusion models incorporating Vietnamese linguistic features, and introduces ViTextCaps, the first large-scale Vietnamese scene-text captioning dataset.

Findings

Cross-modal graph edges are harmful for scene-text fusion.

Phonological fusion improves Vietnamese captioning accuracy.

ViTextCaps enables comprehensive evaluation of Vietnamese scene-text captioning.

Abstract

Scene-text image captioning requires fusing three information streams -- visual features, OCR-detected text, and linguistic knowledge -- to generate descriptions that faithfully integrate text visible in images. Existing fusion approaches treat text as language-agnostic, which fails for Vietnamese: a tonal language where diacritics alter word meaning, OCR errors are pervasive, and word boundaries are ambiguous. We argue that Vietnamese scene-text captioning demands \textit{linguistically informed multimodal fusion}, where language-specific structural knowledge is explicitly incorporated into the fusion mechanism. Motivated from these insights, we propose \textbf{HSTFG} (Heterogeneous Scene-Text Fusion Graph), a general-purpose graph fusion framework with learned spatial attention bias, and show through topology analysis that cross-modal graph edges are harmful for scene-text fusion. Building on this finding, we design \textbf{PhonoSTFG} (Phonological Scene-Text Fusion Graph) which specializes graph-level fusion for Vietnamese linguistic reasoning. To support evaluation, we introduce \textbf{ViTextCaps}, the first large-scale Vietnamese scene-text captioning dataset (\textbf{15{,}729} images with \textbf{74{,}970} captions), with comprehensive linguistic analysis showing that 52.8\% of the vocabulary is at risk of diacritic collision.