Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses

TL;DR

This study compares human and large language model perceptions of English spelling variations on Twitter, revealing strong overall correlations but notable differences in specific social attribute ratings.

Contribution

It introduces a sociolinguistic framework to evaluate how LLMs and humans perceive spelling variation in social contexts, highlighting areas of alignment and divergence.

Findings

Strong correlation in social attribute ratings between humans and LLMs

Differences in rating distributions across spelling variation types

Insights into LLMs' understanding of social cues in online writing

Abstract

Spelling variation (e.g. funnnn vs. fun) can influence the social perception of texts and their writers: we often have various associations with different forms of writing (is the text informal? does the writer seem young?). In this study, we focus on the social perception of spelling variation in online writing in English and study to what extent this perception is aligned between humans and large language models (LLMs). Building on sociolinguistic methodology, we compare LLM and human ratings on three key social attributes of spelling variation (formality, carefulness, age). We find generally strong correlations in the ratings between humans and LLMs. However, notable differences emerge when we analyze the distribution of ratings and when comparing between different types of spelling variation.