I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews

TL;DR

This paper introduces a multilingual dataset for counterfactual detection in product reviews, covering English, German, and Japanese, and evaluates models showing the importance of language-specific approaches.

Contribution

It provides a high-quality, multilingual CFD dataset from e-commerce reviews, and analyzes the effectiveness of different models and translation methods for this task.

Findings

Models are robust against cue phrase biases

Multilingual dataset is compatible with existing data

Machine translation performs poorly for multilingual CFD

Abstract

Counterfactual statements describe events that did not or cannot take place. We consider the problem of counterfactual detection (CFD) in product reviews. For this purpose, we annotate a multilingual CFD dataset from Amazon product reviews covering counterfactual statements written in English, German, and Japanese languages. The dataset is unique as it contains counterfactuals in multiple languages, covers a new application area of e-commerce reviews, and provides high quality professional annotations. We train CFD models using different text representation methods and classifiers. We find that these models are robust against the selectional biases introduced due to cue phrase-based sentence selection. Moreover, our CFD dataset is compatible with prior datasets and can be merged to learn accurate CFD models. Applying machine translation on English counterfactual examples to create multilingual data performs poorly, demonstrating the language-specificity of this problem, which has been ignored so far.