RMPL: Relation-aware Multi-task Progressive Learning with Stage-wise Training for Multimedia Event Extraction

TL;DR

This paper introduces RMPL, a relation-aware multi-task progressive learning framework that enhances multimedia event extraction by leveraging heterogeneous supervision and stage-wise training, especially effective under low-resource conditions.

Contribution

It presents a novel multi-task progressive learning approach that explicitly models relations and utilizes heterogeneous supervision for improved multimedia event extraction.

Findings

RMPL outperforms existing methods on the M2E2 benchmark.

Stage-wise training improves event and argument extraction accuracy.

Incorporating heterogeneous supervision enhances multimodal grounding.

Abstract

Multimedia Event Extraction (MEE) aims to identify events and their arguments from documents that contain both text and images. It requires grounding event semantics across different modalities. Progress in MEE is limited by the lack of annotated training data. M2E2 is the only established benchmark, but it provides annotations only for evaluation. This makes direct supervised training impractical. Existing methods mainly rely on cross-modal alignment or inference-time prompting with Vision--Language Models (VLMs). These approaches do not explicitly learn structured event representations and often produce weak argument grounding in multimodal settings. To address these limitations, we propose RMPL, a Relation-aware Multi-task Progressive Learning framework for MEE under low-resource conditions. RMPL incorporates heterogeneous supervision from unimodal event extraction and multimedia relation extraction with stage-wise training. The model is first trained with a unified schema to learn shared event-centric representations across modalities. It is then fine-tuned for event mention identification and argument role extraction using mixed textual and visual data. Experiments on the M2E2 benchmark with multiple VLMs show consistent improvements across different modality settings.