SocialVeil: Probing Social Intelligence of Language Agents under Communication Barriers

TL;DR

SocialVeil introduces a realistic social interaction environment for language models, simulating communication barriers like semantic vagueness and cultural mismatch, to better evaluate their social intelligence in imperfect settings.

Contribution

This paper presents SocialVeil, a novel environment with barrier simulations and evaluation metrics, to assess LLM social intelligence under communication disruptions, addressing limitations of prior idealized benchmarks.

Findings

Barriers significantly reduce mutual understanding by over 45%.

Confusion levels increase by nearly 50% under communication barriers.

Human evaluations confirm the fidelity of simulated barriers.

Abstract

Large language models (LLMs) are increasingly evaluated in interactive environments to test their social intelligence. However, existing benchmarks often assume idealized communication between agents, limiting our ability to diagnose whether LLMs can maintain and repair interactions in more realistic, imperfect settings. To close this gap, we present \textsc{SocialVeil}, a social learning environment that can simulate social interaction under cognitive-difference-induced communication barriers. Grounded in a systematic literature review of communication challenges in human interaction, \textsc{SocialVeil} introduces three representative types of such disruption, \emph{semantic vagueness}, \emph{sociocultural mismatch}, and \emph{emotional interference}. We also introduce two barrier-aware evaluation metrics, \emph{unresolved confusion} and \emph{mutual understanding}, to evaluate interaction quality under impaired communication. Experiments across 720 scenarios and four frontier LLMs show that barriers consistently impair performance, with mutual understanding reduced by over 45\% on average, and confusion elevated by nearly 50\%. Human evaluations validate the fidelity of these simulated barriers (ICC$\approx$0.78, Pearson r$\approx$0.80). We further demonstrate that adaptation strategies (Repair Instruction and Interactive learning) only have a modest effect far from barrier-free performance. This work takes a step toward bringing social interaction environments closer to real-world communication, opening opportunities for exploring the social intelligence of LLM agents.