Kernel Taylor-Based Value Function Approximation for Continuous-State Markov Decision Processes

TL;DR

This paper introduces a kernel-based policy iteration method for continuous-state MDPs that does not require known transition models, using Taylor expansion and PDE approximation to improve planning efficiency.

Contribution

It presents a novel kernel Taylor-based approach that eliminates the need for explicit transition models in continuous-state MDPs, enabling more practical policy iteration.

Findings

Outperforms baseline methods in simulations

Efficient policy evaluation via linear systems

Effective in both simplified and realistic scenarios

Abstract

We propose a principled kernel-based policy iteration algorithm to solve the continuous-state Markov Decision Processes (MDPs). In contrast to most decision-theoretic planning frameworks, which assume fully known state transition models, we design a method that eliminates such a strong assumption, which is oftentimes extremely difficult to engineer in reality. To achieve this, we first apply the second-order Taylor expansion of the value function. The Bellman optimality equation is then approximated by a partial differential equation, which only relies on the first and second moments of the transition model. By combining the kernel representation of value function, we then design an efficient policy iteration algorithm whose policy evaluation step can be represented as a linear system of equations characterized by a finite set of supporting states. We have validated the proposed method through extensive simulations in both simplified and realistic planning scenarios, and the experiments show that our proposed approach leads to a much superior performance over several baseline methods.