DynaSent: A Dynamic Benchmark for Sentiment Analysis

TL;DR

DynaSent is a large, evolving benchmark for three-way sentiment analysis that combines human and model-in-the-loop data creation, aiming to challenge and improve sentiment models over time.

Contribution

We introduce DynaSent, a dynamic, high-quality sentiment benchmark that evolves through successive model iterations, incorporating human validation and focusing on category coherence.

Findings

DynaSent contains 121,634 validated sentences.

The Neutral category in DynaSent is more coherent than in other benchmarks.

Training models from scratch each round improves performance.

Abstract

We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis. DynaSent combines naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation. DynaSent has a total of 121,634 sentences, each validated by five crowdworkers, and its development and test splits are designed to produce chance performance for even the best models we have been able to develop; when future models solve this task, we will use them to create DynaSent version 2, continuing the dynamic evolution of this benchmark. Here, we report on the dataset creation effort, focusing on the steps we took to increase quality and reduce artifacts. We also present evidence that DynaSent's Neutral category is more coherent than the comparable category in other benchmarks, and we motivate training models from scratch for each round over successive fine-tuning.