SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research

TL;DR

SuperTweetEval is a comprehensive benchmark designed to unify and evaluate NLP models across diverse social media tasks, highlighting ongoing challenges despite recent advances in language modeling.

Contribution

The paper introduces SuperTweetEval, a novel unified benchmark for social media NLP, combining multiple tasks and datasets to facilitate comprehensive model evaluation.

Findings

Social media NLP remains challenging despite recent language model improvements.

Benchmarking shows varied model performance across different social media tasks.

SuperTweetEval enables fair comparison of models on heterogeneous social media datasets.

Abstract

Despite its relevance, the maturity of NLP for social media pales in comparison with general-purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.