Variance Adjusted Actor Critic Algorithms

TL;DR

This paper introduces a novel actor-critic framework for Markov Decision Processes that optimizes a variance-adjusted expected return, with proven convergence to local optima using linear function approximation.

Contribution

It extends compatible features to variance-adjusted objectives and provides an episodic actor-critic algorithm with convergence guarantees.

Findings

Converges almost surely to local optima

Extends compatible features to variance-adjusted setting

Demonstrates effectiveness of the proposed algorithm

Abstract

We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost surely to a locally optimal point of the objective function.