Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis

TL;DR

This paper evaluates ChatGPT's capabilities in causal text mining across diverse datasets, revealing its strengths as a starting point but also its limitations compared to specialized models, especially with complex causality and domain adaptation.

Contribution

The study introduces a comprehensive benchmark and evaluation framework for ChatGPT's causal text mining, highlighting its performance, limitations, and future challenges in this domain.

Findings

ChatGPT is a good starting point for causal text mining.

Specialized models outperform ChatGPT with sufficient training data.

ChatGPT tends to falsely identify non-causal sequences as causal.

Abstract

Causality is fundamental in human cognition and has drawn attention in diverse research fields. With growing volumes of textual data, discerning causalities within text data is crucial, and causal text mining plays a pivotal role in extracting meaningful patterns. This study conducts comprehensive evaluations of ChatGPT's causal text mining capabilities. Firstly, we introduce a benchmark that extends beyond general English datasets, including domain-specific and non-English datasets. We also provide an evaluation framework to ensure fair comparisons between ChatGPT and previous approaches. Finally, our analysis outlines the limitations and future challenges in employing ChatGPT for causal text mining. Specifically, our analysis reveals that ChatGPT serves as a good starting point for various datasets. However, when equipped with a sufficient amount of training data, previous models still surpass ChatGPT's performance. Additionally, ChatGPT suffers from the tendency to falsely recognize non-causal sequences as causal sequences. These issues become even more pronounced with advanced versions of the model, such as GPT-4. In addition, we highlight the constraints of ChatGPT in handling complex causality types, including both intra/inter-sentential and implicit causality. The model also faces challenges with effectively leveraging in-context learning and domain adaptation. We release our code to support further research and development in this field.