Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues

TL;DR

This paper systematically evaluates the multifaceted negotiation capabilities of large language models, especially GPT-4, across various dialogue scenarios, highlighting strengths and challenges in strategic reasoning and contextual understanding.

Contribution

It provides a comprehensive evaluation framework for LLMs in negotiation, revealing their strengths and limitations in complex dialogue tasks.

Findings

GPT-4 outperforms other models in many negotiation tasks

LLMs face challenges in subjective assessments and strategic response generation

The study offers insights for developing better AI negotiation agents

Abstract

A successful negotiation requires a range of capabilities, including comprehension of the conversation context, Theory-of-Mind (ToM) skills to infer the partner's motives, strategic reasoning, and effective communication, making it challenging for automated systems. Despite the remarkable performance of LLMs in various NLP tasks, there is no systematic evaluation of their capabilities in negotiation. Such an evaluation is critical for advancing AI negotiation agents and negotiation research, ranging from designing dialogue systems to providing pedagogical feedback and scaling up data collection practices. This work aims to systematically analyze the multifaceted capabilities of LLMs across diverse dialogue scenarios throughout the stages of a typical negotiation interaction. Our analysis highlights GPT-4's superior performance in many tasks while identifying specific challenges, such as making subjective assessments and generating contextually appropriate, strategically advantageous responses.