Auditing Support Strategies in LLMs through Grounded Multi-Turn Social Simulation

TL;DR

This paper presents a multi-turn social simulation framework to evaluate how large language models provide social support, revealing dynamics invisible to single-turn assessments.

Contribution

It introduces a novel multi-turn simulation approach and uses SSBC coding to analyze support strategies and their relation to user distress in LLMs.

Findings

Support strategies decline as estimated distress increases.

Support composition shifts systematically with user distress across models.

Community context influences support behavior more than demographic factors.

Abstract

When users seek social support from chatbots, they disclose their situation gradually, yet most evaluations of supportive LLMs rely on single-turn, fully specified prompts. We introduce a multi-turn simulation framework that closes this gap. Support-seeking narratives from five Reddit communities are decomposed into ordered fragments and revealed turn by turn to a language model. Each response is coded with the Social Support Behavior Code (SSBC), an established multi-label taxonomy that captures the composition of support, rather than a single quality score. To ask whether support choices track the model's own construal of user distress, we use linear probes on hidden representations to estimate this internal signal without altering the generation context. Across two mid-scale models (Llama-3.1-8B, OLMo-3-7B) and more than 6,200 turns, support composition shifts systematically with estimated distress: teaching declines as estimated distress rises, a finding that replicates across architectures, while increases in affective and esteem-oriented strategies (such as validation) are suggestive but model-specific and rest on noisier annotations. Community context independently shapes behavior, tracking topic and discourse norms rather than demographic categories. These trajectory-level dynamics, invisible to single-turn evaluation, motivate multi-turn auditing frameworks for socially sensitive applications.