Evaluating LLM-persona Generated Distributions for Decision-making

TL;DR

This paper assesses how effectively LLM-generated distributions support decision-making tasks like pricing and assortment optimization, highlighting their practical usefulness especially with limited data and cautioning against relying solely on traditional distribution metrics.

Contribution

It introduces evaluation metrics for LLM-generated distributions in decision-making contexts and demonstrates their effectiveness across multiple canonical problems.

Findings

LLM-generated distributions are useful in low-data scenarios.

Decision-agnostic metrics like Wasserstein distance can be misleading.

LLMs can effectively simulate consumer preferences for decision support.

Abstract

LLMs can generate a wealth of data, ranging from simulated personas imitating human valuations and preferences, to demand forecasts based on world knowledge. But how well do such LLM-generated distributions support downstream decision-making? For example, when pricing a new product, a firm could prompt an LLM to simulate how much consumers are willing to pay based on a product description, but how useful is the resulting distribution for optimizing the price? We refer to this approach as LLM-SAA, in which an LLM is used to construct an estimated distribution and the decision is then optimized under that distribution. In this paper, we study metrics to evaluate the quality of these LLM-generated distributions, based on the decisions they induce. Taking three canonical decision-making problems (assortment optimization, pricing, and newsvendor) as examples, we find that LLM-generated distributions are practically useful, especially in low-data regimes. We also show that decision-agnostic metrics such as Wasserstein distance can be misleading when evaluating these distributions for decision-making.