Reproducibility Study of Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation

TL;DR

This study reproduces and extends previous work on LLM-based negotiation, analyzing model performance, communication dynamics, and fairness, revealing that larger open-weight models can rival proprietary models and that single-agent approaches often suffice.

Contribution

It validates prior findings across diverse models, introduces a communication-free baseline, and offers new insights into model performance, fairness, and privacy in negotiation tasks.

Findings

Larger open-weight models approach proprietary model performance.

Single-agent approaches often match multi-agent negotiation results.

Smaller models (<10B) struggle with format and coherence.

Abstract

This paper presents a reproducibility study and extension of "Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation." We validate the original findings using a range of open-weight models (1.5B-70B parameters) and GPT-4o Mini while introducing several novel contributions. We analyze the Pareto front of the games, propose a communication-free baseline to test whether successful negotiations are possible without agent interaction, evaluate recent small language models' performance, analyze structural information leakage in model responses, and implement an inequality metric to assess negotiation fairness. Our results demonstrate that smaller models (<10B parameters) struggle with format adherence and coherent responses, but larger open-weight models can approach proprietary model performance. Additionally, in many scenarios, single-agent approaches can achieve comparable results to multi-agent negotiations, challenging assumptions about the necessity of agent communication to perform well on the benchmark. This work also provides insights into the accessibility, fairness, environmental impact, and privacy considerations of LLM-based negotiation systems.