Overview of CLEF 2019 Lab ProtestNews: Extracting Protests from News in a Cross-context Setting

TL;DR

This paper overviews the CLEF 2019 ProtestNews lab, which focused on extracting protest-related information from news articles across different countries, highlighting the challenges and neural network performance.

Contribution

It introduces a cross-country protest information extraction task and evaluates neural network approaches in a multilingual, cross-context setting.

Findings

Neural networks achieved the best results.

Performance drops significantly in cross-country settings.

The lab attracted 58 teams with diverse approaches.

Abstract

We present an overview of the CLEF-2019 Lab ProtestNews on Extracting Protests from News in the context of generalizable natural language processing. The lab consists of document, sentence, and token level information classification and extraction tasks that were referred as task 1, task 2, and task 3 respectively in the scope of this lab. The tasks required the participants to identify protest relevant information from English local news at one or more aforementioned levels in a cross-context setting, which is cross-country in the scope of this lab. The training and development data were collected from India and test data was collected from India and China. The lab attracted 58 teams to participate in the lab. 12 and 9 of these teams submitted results and working notes respectively. We have observed neural networks yield the best results and the performance drops significantly for majority of the submissions in the cross-country setting, which is China.