Reasoning and the Trusting Behavior of DeepSeek and GPT: An Experiment Revealing Hidden Fault Lines in Large Language Models

TL;DR

This paper investigates how different large language models exhibit varying trusting behaviors in game-theoretic scenarios, revealing hidden fault lines that impact their reliability in high-stakes applications.

Contribution

It introduces a behavioral economics framework to analyze trust dynamics in LLMs, highlighting differences between models like OpenAI's and DeepSeek's in strategic trust behavior.

Findings

OpenAI's models show a collapse in trust behavior under certain conditions.

DeepSeek's models demonstrate more sophisticated, forward-looking trust strategies.

Relying solely on narrow performance benchmarks can be misleading for high-stakes deployment.

Abstract

When encountering increasingly frequent performance improvements or cost reductions from a new large language model (LLM), developers of applications leveraging LLMs must decide whether to take advantage of these improvements or stay with older tried-and-tested models. Low perceived switching frictions can lead to choices that do not consider more subtle behavior changes that the transition may induce. Our experiments use a popular game-theoretic behavioral economics model of trust to show stark differences in the trusting behavior of OpenAI's and DeepSeek's models. We highlight a collapse in the economic trust behavior of the o1-mini and o3-mini models as they reconcile profit-maximizing and risk-seeking with future returns from trust, and contrast it with DeepSeek's more sophisticated and profitable trusting behavior that stems from an ability to incorporate deeper concepts like forward planning and theory-of-mind. As LLMs form the basis for high-stakes commercial systems, our results highlight the perils of relying on LLM performance benchmarks that are too narrowly defined and suggest that careful analysis of their hidden fault lines should be part of any organization's AI strategy.