Evaluation of the Rate of Convergence in the PIA

TL;DR

This paper demonstrates that Howard's Policy Improvement Algorithm exhibits quadratic local convergence in a general setting, explaining its rapid convergence observed in controlled diffusion problems, supported by numerical examples.

Contribution

The paper extends previous results to show quadratic convergence of the policy improvement algorithm in a more general framework, providing theoretical and numerical evidence.

Findings

Quadratic local convergence of the algorithm is established.

Numerical experiments confirm rapid convergence to the solution.

Theoretical results explain the observed fast convergence in practice.

Abstract

Folklore says that Howard's Policy Improvement Algorithm converges extraordinarily fast, even for controlled diffusion settings. In a previous paper, we proved that approximations of the solution of a particular parabolic partial differential equation obtained via the policy improvement algorithm show a quadratic local convergence. In this paper, we show that we obtain the same rate of convergence of the algorithm in a more general setup. This provides some explanation as to why the algorithm converges fast. We provide an example by solving a semilinear elliptic partial differential equation numerically by applying the algorithm and check how the approximations converge to the analytic solution.