Unimodal and Multimodal Representation Training for Relation Extraction

TL;DR

This paper evaluates the predictive power of text, layout, and visual modalities in relation extraction tasks, highlighting the importance of shared representations and the potential of unimodal approaches.

Contribution

It systematically assesses the contribution of each modality in multimodal relation extraction, revealing the significance of text and layout, and exploring unimodal and multimodal training strategies.

Findings

Text is the most important predictor of entity relations.

Layout geometry is highly predictive and can be used unimodally.

Visual information can enhance performance in certain circumstances.

Abstract

Multimodal integration of text, layout and visual information has achieved SOTA results in visually rich document understanding (VrDU) tasks, including relation extraction (RE). However, despite its importance, evaluation of the relative predictive capacity of these modalities is less prevalent. Here, we demonstrate the value of shared representations for RE tasks by conducting experiments in which each data type is iteratively excluded during training. In addition, text and layout data are evaluated in isolation. While a bimodal text and layout approach performs best (F1=0.684), we show that text is the most important single predictor of entity relations. Additionally, layout geometry is highly predictive and may even be a feasible unimodal approach. Despite being less effective, we highlight circumstances where visual information can bolster performance. In total, our results demonstrate the efficacy of training joint representations for RE.