STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models

TL;DR

This paper introduces STEER-ME, a comprehensive benchmark for evaluating large language models' microeconomic reasoning across diverse tasks, domains, and perspectives, using an innovative auto-STEER data generation protocol.

Contribution

It presents a novel taxonomy of microeconomic reasoning elements and an automated data generation method to create diverse, unbiased evaluation questions for LLMs.

Findings

Large models outperform smaller ones in microeconomic reasoning.

Prompting strategies significantly affect model performance.

Auto-STEER enables scalable and unbiased benchmark creation.

Abstract

How should one judge whether a given large language model (LLM) can reliably perform economic reasoning? Most existing LLM benchmarks focus on specific applications and fail to present the model with a rich variety of economic tasks. A notable exception is Raman et al. [2024], who offer an approach for comprehensively benchmarking strategic decision-making; however, this approach fails to address the non-strategic settings prevalent in microeconomics, such as supply-and-demand analysis. We address this gap by taxonomizing microeconomic reasoning into $58$ distinct elements, focusing on the logic of supply and demand, each grounded in up to $10$ distinct domains, $5$ perspectives, and $3$ types. The generation of benchmark data across this combinatorial space is powered by a novel LLM-assisted data generation protocol that we dub auto-STEER, which generates a set of questions by adapting handwritten templates to target new domains and perspectives. Because it offers an automated way of generating fresh questions, auto-STEER mitigates the risk that LLMs will be trained to over-fit evaluation benchmarks; we thus hope that it will serve as a useful tool both for evaluating and fine-tuning models for years to come. We demonstrate the usefulness of our benchmark via a case study on $27$ LLMs, ranging from small open-source models to the current state of the art. We examined each model's ability to solve microeconomic problems across our whole taxonomy and present the results across a range of prompting strategies and scoring metrics.