LLMs as mediators: Can they diagnose conflicts accurately?

TL;DR

This study evaluates GPT 3.5 and GPT 4's ability to diagnose conflict sources, revealing they understand causal and moral distinctions but tend to misjudge their prevalence, with GPT 4 performing better.

Contribution

First empirical assessment of LLMs' capacity to identify causal versus moral disagreements in conflict mediation tasks.

Findings

Both LLMs understand causal and moral distinctions similarly to humans.

LLMs tend to overestimate causal disagreements and underestimate moral disagreements.

GPT 4 outperforms GPT 3.5 and humans in diagnosing conflict sources.

Abstract

Prior research indicates that to be able to mediate conflict, observers of disagreements between parties must be able to reliably distinguish the sources of their disagreement as stemming from differences in beliefs about what is true (causality) vs. differences in what they value (morality). In this paper, we test if OpenAI's Large Language Models GPT 3.5 and GPT 4 can perform this task and whether one or other type of disagreement proves particularly challenging for LLM's to diagnose. We replicate study 1 in Ko\c{c}ak et al. (2003), which employes a vignette design, with OpenAI's GPT 3.5 and GPT 4. We find that both LLMs have similar semantic understanding of the distinction between causal and moral codes as humans and can reliably distinguish between them. When asked to diagnose the source of disagreement in a conversation, both LLMs, compared to humans, exhibit a tendency to overestimate the extent of causal disagreement and underestimate the extent of moral disagreement in the moral misalignment condition. This tendency is especially pronounced for GPT 4 when using a proximate scale that relies on concrete language specific to an issue. GPT 3.5 does not perform as well as GPT4 or humans when using either the proximate or the distal scale. The study provides a first test of the potential for using LLMs to mediate conflict by diagnosing the root of disagreements in causal and evaluative codes.