Inflation Attitudes of Large Language Models

TL;DR

This paper evaluates GPT-3.5-turbo's ability to perceive and predict inflation trends, comparing its responses to survey data and official stats, revealing strengths in short-term tracking and key perception patterns but limitations in modeling consumer inflation.

Contribution

It introduces a novel method to analyze LLMs' inflation perceptions using a quasi-experimental design and Shapley value decomposition, providing insights into model drivers and sensitivities.

Findings

GPT tracks survey projections and official statistics at short horizons.

GPT replicates key household inflation perception patterns for income, housing, and social class.

GPT shows heightened sensitivity to food inflation similar to humans but lacks a consistent inflation model.

Abstract

This paper investigates the ability of Large Language Models (LLMs), specifically GPT-3.5-turbo (GPT), to form inflation perceptions and expectations based on macroeconomic price signals. We compare the LLM's output to household survey data and official statistics, mimicking the information set and demographic characteristics of the Bank of England's Inflation Attitudes Survey (IAS). Our quasi-experimental design exploits the timing of GPT's training cut-off in September 2021 which means it has no knowledge of the subsequent UK inflation surge. We find that GPT tracks aggregate survey projections and official statistics at short horizons. At a disaggregated level, GPT replicates key empirical regularities of households' inflation perceptions, particularly for income, housing tenure, and social class. A novel Shapley value decomposition of LLM outputs suited for the synthetic survey setting provides well-defined insights into the drivers of model outputs linked to prompt content. We find that GPT demonstrates a heightened sensitivity to food inflation information similar to that of human respondents. However, we also find that it lacks a consistent model of consumer price inflation. More generally, our approach could be used to evaluate the behaviour of LLMs for use in the social sciences, to compare different models, or to assist in survey design.