Convex Q Learning in a Stochastic Environment: Extended Version

TL;DR

This paper develops a convex formulation of Q-learning for Markov decision processes with function approximation, providing new algorithms with convergence guarantees and applications to inventory control.

Contribution

It introduces a convex relaxation of Q-learning, analyzes its properties, and proposes convergent, model-free algorithms with variance reduction techniques.

Findings

Bounded solutions under simple basis function conditions

Convergence of the proposed algorithms with rate analysis

Application demonstrated on inventory control problem

Abstract

The paper introduces the first formulation of convex Q-learning for Markov decision processes with function approximation. The algorithms and theory rest on a relaxation of a dual of Manne's celebrated linear programming characterization of optimal control. The main contributions firstly concern properties of the relaxation, described as a deterministic convex program: we identify conditions for a bounded solution, and a significant relationship between the solution to the new convex program, and the solution to standard Q-learning. The second set of contributions concern algorithm design and analysis: (i) A direct model-free method for approximating the convex program for Q-learning shares properties with its ideal. In particular, a bounded solution is ensured subject to a simple property of the basis functions; (ii) The proposed algorithms are convergent and new techniques are introduced to obtain the rate of convergence in a mean-square sense; (iii) The approach can be generalized to a range of performance criteria, and it is found that variance can be reduced by considering ``relative'' dynamic programming equations; (iv) The theory is illustrated with an application to a classical inventory control problem.