MAVEN-Fact: A Large-scale Event Factuality Detection Dataset

TL;DR

MAVEN-Fact introduces the largest high-quality dataset for event factuality detection, enabling better understanding of event knowledge and highlighting the differing impacts of event arguments on models.

Contribution

It provides a large-scale, high-quality EFD dataset based on MAVEN, supporting new analyses and applications in event factuality detection.

Findings

MAVEN-Fact is challenging for models and LLMs.

Event arguments aid fine-tuned models but not LLMs.

Event factuality detection can reduce hallucinations in LLMs.

Abstract

Event Factuality Detection (EFD) task determines the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility, which is essential for faithfully understanding and utilizing event knowledge. However, due to the lack of high-quality large-scale data, event factuality detection is under-explored in event understanding research, which limits the development of EFD community. To address these issues and provide faithful event understanding, we introduce MAVEN-Fact, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-Fact includes factuality annotations of 112,276 events, making it the largest EFD dataset. Extensive experiments demonstrate that MAVEN-Fact is challenging for both conventional fine-tuned models and large language models (LLMs). Thanks to the comprehensive annotations of event arguments and relations in MAVEN, MAVEN-Fact also supports some further analyses and we find that adopting event arguments and relations helps in event factuality detection for fine-tuned models but does not benefit LLMs. Furthermore, we preliminarily study an application case of event factuality detection and find it helps in mitigating event-related hallucination in LLMs. Our dataset and codes can be obtained from \url{https://github.com/lcy2723/MAVEN-FACT}