Debate as Optimization: Adaptive Conformal Prediction and Diverse Retrieval for Event Extraction

TL;DR

This paper introduces a debate-based optimization framework for event extraction using large language models, incorporating novel modules for information retrieval and confidence estimation to improve accuracy without parameter tuning.

Contribution

It presents a new multi-agent debate system with two modules, DRAG and AdaCP, that enhance event extraction performance in a tuning-free manner.

Findings

Achieved an 18.1% and 17.8% reduction in performance gap on ACE05.

Achieved a 17.9% and 15.2% reduction on CASIE datasets.

Demonstrated effectiveness of debate-based optimization for event extraction.

Abstract

We propose a multi-agent debate as optimization (DAO) system for event extraction, where the primary objective is to iteratively refine the large language models (LLMs) outputs through debating without parameter tuning. In DAO, we introduce two novel modules: the Diverse-RAG (DRAG) module and the Adaptive Conformal Prediction (AdaCP) module. DRAG systematically retrieves supporting information that best fits the debate discussion, while AdaCP enhances the accuracy and reliability of event extraction by effectively rejecting less promising answers. Experimental results demonstrate a significant reduction in the performance gap between supervised approaches and tuning-free LLM-based methods by 18.1% and 17.8% on ACE05 and 17.9% and 15.2% on CASIE for event detection and argument extraction respectively.