Dynamic Programming: From Local Optimality to Global Optimality

TL;DR

This paper explores conditions under which local optimality in dynamic programming guarantees global optimality, with implications for large-scale policy algorithms and neural network applications.

Contribution

It provides sufficient conditions linking local and global optimality in dynamic programming, extending understanding to neural network-based policy methods.

Findings

Established conditions for local to global optimality transition.

Applied results to neural network-based policy optimization.

Demonstrated implications for large-scale dynamic programming algorithms.

Abstract

In the theory of dynamic programming, an optimal policy is a policy whose lifetime value dominates that of all other policies from every possible initial condition in the state space. This raises a natural question: when does optimality from a single state imply optimality from every state? Working in a general setting, we provide sufficient conditions for this property that relate to reachability and irreducibility. Our results have significant implications for modern policy-based algorithms used to solve large-scale dynamic programs. We illustrate our findings by applying them to an optimal savings problem via an algorithm that implements gradient ascent in a policy space constructed from neural networks.