Identifying economic narratives in large text corpora -- An integrated approach using Large Language Models

TL;DR

This paper evaluates the use of GPT-4o, a large language model, for extracting economic narratives from news articles, comparing its performance to expert annotations and discussing implications for future economic research.

Contribution

It demonstrates the potential and limitations of LLMs like GPT-4o in extracting economic narratives, offering a new approach beyond traditional NLP pipelines.

Findings

GPT-4o can extract structured economic narratives

GPT-4o performs below expert level on complex documents

Provides guidance for future LLM applications in economics

Abstract

As interest in economic narratives has grown in recent years, so has the number of pipelines dedicated to extracting such narratives from texts. Pipelines often employ a mix of state-of-the-art natural language processing techniques, such as BERT, to tackle this task. While effective on foundational linguistic operations essential for narrative extraction, such models lack the deeper semantic understanding required to distinguish extracting economic narratives from merely conducting classic tasks like Semantic Role Labeling. Instead of relying on complex model pipelines, we evaluate the benefits of Large Language Models (LLMs) by analyzing a corpus of Wall Street Journal and New York Times newspaper articles about inflation. We apply a rigorous narrative definition and compare GPT-4o outputs to gold-standard narratives produced by expert annotators. Our results suggests that GPT-4o is capable of extracting valid economic narratives in a structured format, but still falls short of expert-level performance when handling complex documents and narratives. Given the novelty of LLMs in economic research, we also provide guidance for future work in economics and the social sciences that employs LLMs to pursue similar objectives.