OTCR: Optimal Transmission, Compression and Representation for Multimodal Information Extraction

TL;DR

OTCR introduces a task-centric, interpretable framework for multimodal information extraction that selectively fuses text and visual cues, improving efficiency and reducing redundancy in document AI applications.

Contribution

It proposes a novel two-stage approach using optimal transport and information bottleneck to enhance multimodal fusion with controllable, task-specific representations.

Findings

Achieves state-of-the-art results on FUNSD and XFUND datasets.

Reduces modality redundancy and enhances task signal strength.

Provides an interpretable, information-theoretic fusion paradigm.

Abstract

Multimodal Information Extraction (MIE) requires fusing text and visual cues from visually rich documents. While recent methods have advanced multimodal representation learning, most implicitly assume modality equivalence or treat modalities in a largely uniform manner, still relying on generic fusion paradigms. This often results in indiscriminate incorporation of multimodal signals and insufficient control over task-irrelevant redundancy, which may in turn limit generalization. We revisit MIE from a task-centric view: text should dominate, vision should selectively support. We present OTCR, a two-stage framework. First, Cross-modal Optimal Transport (OT) yields sparse, probabilistic alignments between text tokens and visual patches, with a context-aware gate controlling visual injection. Second, a Variational Information Bottleneck (VIB) compresses fused features, filtering task-irrelevant noise to produce compact, task-adaptive representations. On FUNSD, OTCR achieves 91.95% SER and 91.13% RE, while on XFUND (ZH), it reaches 91.09% SER and 94.20% RE, demonstrating competitive performance across datasets. Feature-level analyses further confirm reduced modality redundancy and strengthened task signals. This work offers an interpretable, information-theoretic paradigm for controllable multimodal fusion in document AI.