SoMe: A Realistic Benchmark for LLM-based Social Media Agents

TL;DR

SoMe is a comprehensive benchmark designed to evaluate the capabilities of LLM-based social media agents across diverse, realistic social media tasks, highlighting current limitations of existing models.

Contribution

This paper introduces SoMe, the first versatile and realistic benchmark for assessing social media agents powered by LLMs across multiple tasks and datasets.

Findings

Current LLMs struggle with social media agent tasks.

SoMe provides a challenging environment for future model development.

Extensive analysis reveals limitations of both open-source and closed-source LLMs.

Abstract

Intelligent agents powered by large language models (LLMs) have recently demonstrated impressive capabilities and gained increasing popularity on social media platforms. While LLM agents are reshaping the ecology of social media, there exists a current gap in conducting a comprehensive evaluation of their ability to comprehend media content, understand user behaviors, and make intricate decisions. To address this challenge, we introduce SoMe, a pioneering benchmark designed to evaluate social media agents equipped with various agent tools for accessing and analyzing social media data. SoMe comprises a diverse collection of 8 social media agent tasks, 9,164,284 posts, 6,591 user profiles, and 25,686 reports from various social media platforms and external websites, with 17,869 meticulously annotated task queries. Compared with the existing datasets and benchmarks for social media tasks, SoMe is the first to provide a versatile and realistic platform for LLM-based social media agents to handle diverse social media tasks. By extensive quantitative and qualitative analysis, we provide the first overview insight into the performance of mainstream agentic LLMs in realistic social media environments and identify several limitations. Our evaluation reveals that both the current closed-source and open-source LLMs cannot handle social media agent tasks satisfactorily. SoMe provides a challenging yet meaningful testbed for future social media agents. Our code and data are available at https://github.com/LivXue/SoMe