Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media

TL;DR

This paper introduces a question-based evaluation framework to assess LLMs' ability to process and analyze large-scale social media text data, revealing performance limitations related to input size and task complexity.

Contribution

The paper presents a novel benchmark with 470 questions for evaluating LLMs on social media data analysis, highlighting their current limitations in handling large inputs and complex tasks.

Findings

Performance declines with increased input size beyond 500 instances.

Accuracy drops in multi-label and target-dependent tasks.

Numerical and quantitative tasks are particularly challenging for LLMs.

Abstract

LLMs have demonstrated exceptional proficiency in a wide range of NLP tasks. However, a notable gap remains in practical data analysis scenarios, particularly when LLMs are required to process long sequences of unstructured documents, such as news feeds or, as specifically addressed in this paper, social media posts. To empirically assess the effectiveness of LLMs in this setting, we introduce a question-based evaluation framework comprising 470 manually curated questions designed to evaluate LLMs' semantic understanding and reasoning abilities over aggregated text data. We apply our benchmark on diverse Twitter datasets covering various NLP tasks, including sentiment analysis, hate speech detection, and emotion recognition. Our results reveal that the performance depends heavily on input scale and the complexity of the data sources, declining noticeably in multi-label or target-dependent scenarios. In addition, as task complexity increases, performance drops progressively from basic semantic existence identification to more demanding operations such as comparison, counting, and calculation. Furthermore, as the input size grows beyond 500 instances, we identify a common limitation across LLMs, particularly Open-weights models: performance degrades substantially, especially on numerical tasks. These findings highlight critical architectural bottlenecks in current LLMs for performing rigorous quantitative analysis over large text collections.