Causality Detection using Multiple Annotation Decisions

TL;DR

This paper explores causality detection in protest news using large language models with customized loss functions, achieving high accuracy on the Causal News Corpus dataset.

Contribution

It introduces a novel approach of leveraging annotation information with customized loss functions in large language models for causality detection.

Findings

BERT-based models with refined loss functions outperform others.

Achieved an F1 score of 0.8501 on the Causal News Corpus.

Customized cross-entropy loss improves causality detection accuracy.

Abstract

The paper describes the work that has been submitted to the 5th workshop on Challenges and Applications of Automated Extraction of socio-political events from text (CASE 2022). The work is associated with Subtask 1 of Shared Task 3 that aims to detect causality in protest news corpus. The authors used different large language models with customized cross-entropy loss functions that exploit annotation information. The experiments showed that bert-based-uncased with refined cross-entropy outperformed the others, achieving a F1 score of 0.8501 on the Causal News Corpus dataset.