Evaluating LLMs Capabilities Towards Understanding Social Dynamics

TL;DR

This paper critically evaluates the ability of large language models to understand social media dynamics, focusing on language, directionality, and bullying detection, revealing mixed results and the impact of fine-tuning.

Contribution

It provides a comparative analysis of LLMs' social understanding capabilities and highlights the effects of fine-tuning and prompt engineering on performance.

Findings

Fine-tuned LLMs show promise in understanding social media directionality.

Mixed results in paraphrasing and bullying detection tasks.

Fine-tuning and prompt engineering can improve certain social understanding tasks.

Abstract

Social media discourse involves people from different backgrounds, beliefs, and motives. Thus, often such discourse can devolve into toxic interactions. Generative Models, such as Llama and ChatGPT, have recently exploded in popularity due to their capabilities in zero-shot question-answering. Because these models are increasingly being used to ask questions of social significance, a crucial research question is whether they can understand social media dynamics. This work provides a critical analysis regarding generative LLM's ability to understand language and dynamics in social contexts, particularly considering cyberbullying and anti-cyberbullying (posts aimed at reducing cyberbullying) interactions. Specifically, we compare and contrast the capabilities of different large language models (LLMs) to understand three key aspects of social dynamics: language, directionality, and the occurrence of bullying/anti-bullying messages. We found that while fine-tuned LLMs exhibit promising results in some social media understanding tasks (understanding directionality), they presented mixed results in others (proper paraphrasing and bullying/anti-bullying detection). We also found that fine-tuning and prompt engineering mechanisms can have positive effects in some tasks. We believe that a understanding of LLM's capabilities is crucial to design future models that can be effectively used in social applications.