Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View

TL;DR

This paper investigates whether large language model (LLM) agents' hallucinations can reflect human-like irrational social behaviors, proposing a framework to assess and enhance their social intelligence through cognitive biases.

Contribution

It introduces CogMir, a novel multi-LLM framework that leverages hallucination properties to evaluate and improve LLM agents' social cognition and irrational decision-making.

Findings

LLM agents show high consistency with humans in irrational social decisions.

Hallucination properties can be used to assess social biases in LLMs.

CogMir framework is effective for studying social intelligence in LLMs.

Abstract

Large language models (LLMs) have been shown to face hallucination issues due to the data they trained on often containing human bias; whether this is reflected in the decision-making process of LLM Agents remains under-explored. As LLM Agents are increasingly employed in intricate social environments, a pressing and natural question emerges: Can we utilize LLM Agents' systematic hallucinations to mirror human cognitive biases, thus exhibiting irrational social intelligence? In this paper, we probe the irrational behavior among contemporary LLM Agents by melding practical social science experiments with theoretical insights. Specifically, We propose CogMir, an open-ended Multi-LLM Agents framework that utilizes hallucination properties to assess and enhance LLM Agents' social intelligence through cognitive biases. Experimental results on CogMir subsets show that LLM Agents and humans exhibit high consistency in irrational and prosocial decision-making under uncertain conditions, underscoring the prosociality of LLM Agents as social entities and highlighting the significance of hallucination properties. Additionally, the CogMir framework demonstrates its potential as a valuable platform for encouraging more research into the social intelligence of LLM Agents.