A constrained optimization perspective on actor critic algorithms and application to network routing

TL;DR

This paper introduces a new actor-critic algorithm with guaranteed convergence for Markov decision processes, extending it to function approximation and demonstrating its effectiveness in network routing applications.

Contribution

The paper presents a novel actor-critic method based on constrained optimization principles, ensuring convergence and applicability to real-world network routing problems.

Findings

Guaranteed convergence to optimal policy

Effective extension with function approximation

Successful application to network routing

Abstract

We propose a novel actor-critic algorithm with guaranteed convergence to an optimal policy for a discounted reward Markov decision process. The actor incorporates a descent direction that is motivated by the solution of a certain non-linear optimization problem. We also discuss an extension to incorporate function approximation and demonstrate the practicality of our algorithms on a network routing application.