Measuring Your ASTE Models in The Wild: A Diversified Multi-domain Dataset For Aspect Sentiment Triplet Extraction

TL;DR

This paper introduces DMASTE, a new diverse and realistic dataset for Aspect Sentiment Triplet Extraction, enabling better evaluation of models in real-world multi-domain scenarios.

Contribution

The paper presents DMASTE, a manually annotated dataset with diverse reviews, aspect types, and domains, addressing limitations of previous datasets for ASTE research.

Findings

DMASTE is more challenging than existing datasets.

Models show varied performance across domains.

Analysis suggests new directions for ASTE research.

Abstract

Aspect Sentiment Triplet Extraction (ASTE) is widely used in various applications. However, existing ASTE datasets are limited in their ability to represent real-world scenarios, hindering the advancement of research in this area. In this paper, we introduce a new dataset, named DMASTE, which is manually annotated to better fit real-world scenarios by providing more diverse and realistic reviews for the task. The dataset includes various lengths, diverse expressions, more aspect types, and more domains than existing datasets. We conduct extensive experiments on DMASTE in multiple settings to evaluate previous ASTE approaches. Empirical results demonstrate that DMASTE is a more challenging ASTE dataset. Further analyses of in-domain and cross-domain settings provide promising directions for future research. Our code and dataset are available at https://github.com/NJUNLP/DMASTE.