TL;DR
This paper introduces the IndiaPoliceEvents corpus, a new dataset for evaluating event extraction in social science, focusing on police activity during the 2002 Gujarat violence, with corpus-level evaluation methods and baseline model results.
Contribution
The paper presents a novel corpus and evaluation framework for social science event extraction, using natural question-based annotations and corpus-level metrics.
Findings
Baseline zero-shot BERT models achieve moderate performance.
Corpus-level evaluation reveals challenges in event extraction tasks.
Natural question annotations enable unbiased recall assessments.
Abstract
Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
