Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

TL;DR

This paper introduces a structural reinforcement learning method that efficiently solves complex heterogeneous agent macroeconomic models by directly learning equilibrium prices from simulations, bypassing traditional high-dimensional equations.

Contribution

The paper presents a novel SRL approach that replaces the cross-sectional distribution with prices as state variables, enabling fast, global solutions for models with aggregate risk and market-clearing conditions.

Findings

Successfully solves Krusell-Smith, Huggett, and HANK models within minutes.

Sidesteps the Master equation, handling complex market dynamics.

Provides a general, efficient solution method for heterogeneous agent models.

Abstract

We present a new approach to formulating and solving heterogeneous agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let agents learn equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting agents' structural knowledge of their own individual dynamics. Our SRL method yields a general and highly efficient global solution method for heterogeneous agent models that sidesteps the Master equation and handles problems traditional methods struggle with, in particular nontrivial market-clearing conditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which we solve globally within minutes.