Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction

TL;DR

This paper introduces ARIS, a hybrid event extraction system combining self-mixture of agents, a sequence tagger, and LLMs with reflective inference to improve accuracy and recall in event extraction tasks.

Contribution

It presents a novel hybrid framework that integrates structured consensus, confidence filtering, and LLM reflection, advancing the state-of-the-art in event extraction.

Findings

Outperforms existing methods on three benchmark datasets.

Enhances event extraction accuracy and recall.

Demonstrates effectiveness of reflective inference and structured consensus.

Abstract

Event Extraction (EE) involves automatically identifying and extracting structured information about events from unstructured text, including triggers, event types, and arguments. Traditional discriminative models demonstrate high precision but often exhibit limited recall, particularly for nuanced or infrequent events. Conversely, generative approaches leveraging Large Language Models (LLMs) provide higher semantic flexibility and recall but suffer from hallucinations and inconsistent predictions. To address these challenges, we propose Agreement-based Reflective Inference System (ARIS), a hybrid approach combining a Self Mixture of Agents with a discriminative sequence tagger. ARIS explicitly leverages structured model consensus, confidence-based filtering, and an LLM reflective inference module to reliably resolve ambiguities and enhance overall event prediction quality. We further investigate decomposed instruction fine-tuning for enhanced LLM event extraction understanding. Experiments demonstrate our approach outperforms existing state-of-the-art event extraction methods across three benchmark datasets.