Towards Trust Calibration in Socially Interactive Agents: Investigating Gendered Multimodal Behaviors Generation with LLMs

TL;DR

This paper investigates how Large Language Models can generate multimodal behaviors reflecting trustworthiness traits, aiming to improve trust calibration in socially interactive agents through nuanced, trait-aligned behavior generation.

Contribution

It introduces a novel method for automatically generating multimodal behaviors aligned with ability and benevolence traits using LLMs, advancing trust-aware social agent interactions.

Findings

GPT-5.4 produces coherent multimodal behaviors across modalities.

Generated behaviors align with theoretical expectations for ability and benevolence.

Gender stereotypes emerge when gender is specified in prompts.

Abstract

As Socially Interactive Agents (SIAs) become increasingly integrated into daily life, the ability to calibrate user trust to an agent's actual capabilities would help ensure appropriate usage of these agents. In this paper, we explore the capacity of Large Language Models (LLMs) to generate multimodal behaviors (verbal, vocal, gestural, and facial expression modalities) that reflect varying levels of ability and benevolence, two key dimensions of trustworthiness. We propose a novel method for automatically generating behaviors aligned with specific levels of these traits, a first step towards enabling nuanced and trust-calibrated interactions. By analyzing a large dataset of multimodal transcripts generated by LLMs, we demonstrate that GPT-5.4 is able to produce coherent behavior across different modalities (text, intonation, facial expression, and gesture). Using Random Forest feature importance analysis, we show that the generated behaviors align with theoretical expectations for ability and benevolence. However, we also find that when gender is specified in the prompt, LLMs tend to reproduce societal gender stereotypes, associating male agents' behaviors with high ability and female agents' behaviors with high benevolence. To validate our approach, we conducted a user study on Prolific using a within-subjects design. Participants perceived different levels of ability and benevolence in the generated behaviors align with the intended instructions.