Digital Skin, Digital Bias: Uncovering Tone-Based Biases in LLMs and Emoji Embeddings

TL;DR

This study compares bias in skin-tone emoji representations across emoji-specific models and large language models, revealing systemic disparities and biases that could impact social inclusion online.

Contribution

It provides the first large-scale comparison of skin-tone emoji bias in dedicated emoji models versus modern LLMs, highlighting critical performance gaps and societal biases.

Findings

LLMs support skin tone modifiers more robustly than emoji-specific models

Specialized emoji models show significant deficiencies in representing skin tones

Evidence of skewed sentiment and inconsistent meanings across skin tones in emoji representations

Abstract

Skin-toned emojis are crucial for fostering personal identity and social inclusion in online communication. As AI models, particularly Large Language Models (LLMs), increasingly mediate interactions on web platforms, the risk that these systems perpetuate societal biases through their representation of such symbols is a significant concern. This paper presents the first large-scale comparative study of bias in skin-toned emoji representations across two distinct model classes. We systematically evaluate dedicated emoji embedding models (emoji2vec, emoji-sw2v) against four modern LLMs (Llama, Gemma, Qwen, and Mistral). Our analysis first reveals a critical performance gap: while LLMs demonstrate robust support for skin tone modifiers, widely-used specialized emoji models exhibit severe deficiencies. More importantly, a multi-faceted investigation into semantic consistency, representational similarity, sentiment polarity, and core biases uncovers systemic disparities. We find evidence of skewed sentiment and inconsistent meanings associated with emojis across different skin tones, highlighting latent biases within these foundational models. Our findings underscore the urgent need for developers and platforms to audit and mitigate these representational harms, ensuring that AI's role on the web promotes genuine equity rather than reinforcing societal biases.