Agentic AI and Hallucinations

TL;DR

This paper models a market where AI agents buy and resell answers, balancing effort to verify accuracy against the risk of hallucinations, with implications for sectors like law and medicine.

Contribution

It introduces a game-theoretic model of AI markets with reputation effects and analyzes how user preferences influence verification efforts and pricing.

Findings

Effort and price increase with the share of users concerned about accuracy.

Hallucination-sensitive sectors induce more verification effort.

A unique equilibrium exists under nontrivial discounting.

Abstract

We model a competitive market where AI agents buy answers from upstream generative models and resell them to users who differ in how much they value accuracy and in how much they fear hallucinations. Agents can privately exert effort for costly verification to lower hallucination risks. Since interactions halt in the event of a hallucination, the threat of losing future rents disciplines effort. A unique reputational equilibrium exists under nontrivial discounting. The equilibrium effort, and thus the price, increases with the share of users who have high accuracy concerns, implying that hallucination-sensitive sectors, such as law and medicine, endogenously lead to more serious verification efforts in agentic AI markets.