Rethinking the Event Coding Pipeline with Prompt Entailment

TL;DR

PR-ENT introduces a flexible, resource-efficient event coding method using prompt entailment with pre-trained language models, enabling domain experts to craft interpretable prompts and answer candidates for classifying political events from news.

Contribution

The paper presents PR-ENT, a novel event coding approach that leverages prompt entailment and human-in-the-loop design, reducing resource needs while maintaining accuracy.

Findings

PR-ENT achieves competitive accuracy in event classification.

The method is robust to description and template perturbations.

It enables domain experts to craft interpretable prompts.

Abstract

For monitoring crises, political events are extracted from the news. The large amount of unstructured full-text event descriptions makes a case-by-case analysis unmanageable, particularly for low-resource humanitarian aid organizations. This creates a demand to classify events into event types, a task referred to as event coding. Typically, domain experts craft an event type ontology, annotators label a large dataset and technical experts develop a supervised coding system. In this work, we propose PR-ENT, a new event coding approach that is more flexible and resource-efficient, while maintaining competitive accuracy: first, we extend an event description such as "Military injured two civilians'' by a template, e.g. "People were [Z]" and prompt a pre-trained (cloze) language model to fill the slot Z. Second, we select answer candidates Z* = {"injured'', "hurt"...} by treating the event description as premise and the filled templates as hypothesis in a textual entailment task. This allows domain experts to draft the codebook directly as labeled prompts and interpretable answer candidates. This human-in-the-loop process is guided by our interactive codebook design tool. We evaluate PR-ENT in several robustness checks: perturbing the event description and prompt template, restricting the vocabulary and removing contextual information.