Policy Gradient for Continuing Tasks in Non-stationary Markov Decision Processes

TL;DR

This paper develops an online policy gradient algorithm for non-stationary Markov decision processes, enabling continuous learning and adaptation without stationarity assumptions, demonstrated through a navigation task example.

Contribution

It introduces a convergence proof for an online policy gradient method in non-stationary MDPs, expanding applicability to continuous, cyclic tasks.

Findings

The algorithm converges to critical points of the initial value function.

It successfully learns cyclic navigation tasks in non-stationary environments.

Theoretical results are supported by numerical experiments.

Abstract

Reinforcement learning considers the problem of finding policies that maximize an expected cumulative reward in a Markov decision process with unknown transition probabilities. In this paper we consider the problem of finding optimal policies assuming that they belong to a reproducing kernel Hilbert space (RKHS). To that end we compute unbiased stochastic gradients of the value function which we use as ascent directions to update the policy. A major drawback of policy gradient-type algorithms is that they are limited to episodic tasks unless stationarity assumptions are imposed. Hence preventing these algorithms to be fully implemented online, which is a desirable property for systems that need to adapt to new tasks and/or environments in deployment. The main requirement for a policy gradient algorithm to work is that the estimate of the gradient at any point in time is an ascent direction for the initial value function. In this work we establish that indeed this is the case which enables to show the convergence of the online algorithm to the critical points of the initial value function. A numerical example shows the ability of our online algorithm to learn to solve a navigation and surveillance problem, in which an agent must loop between to goal locations. This example corroborates our theoretical findings about the ascent directions of subsequent stochastic gradients. It also shows how the agent running our online algorithm succeeds in learning to navigate, following a continuing cyclic trajectory that does not comply with the standard stationarity assumptions in the literature for non episodic training.