EconNLI: Evaluating Large Language Models on Economics Reasoning

TL;DR

This paper introduces EconNLI, a dataset for evaluating large language models' understanding and reasoning about economic events, revealing current models' limitations in economic reasoning and generation.

Contribution

The paper presents a new dataset, EconNLI, specifically designed to assess LLMs' economic reasoning and knowledge, filling a gap in systematic evaluation in this domain.

Findings

LLMs struggle with economic reasoning tasks

Models often generate incorrect or hallucinated economic events

EconNLI enables targeted evaluation of LLMs in economics

Abstract

Large Language Models (LLMs) are widely used for writing economic analysis reports or providing financial advice, but their ability to understand economic knowledge and reason about potential results of specific economic events lacks systematic evaluation. To address this gap, we propose a new dataset, natural language inference on economic events (EconNLI), to evaluate LLMs' knowledge and reasoning abilities in the economic domain. We evaluate LLMs on (1) their ability to correctly classify whether a premise event will cause a hypothesis event and (2) their ability to generate reasonable events resulting from a given premise. Our experiments reveal that LLMs are not sophisticated in economic reasoning and may generate wrong or hallucinated answers. Our study raises awareness of the limitations of using LLMs for critical decision-making involving economic reasoning and analysis. The dataset and codes are available at https://github.com/Irenehere/EconNLI.