Non-Markovian Reinforcement Learning using Fractional Dynamics

TL;DR

This paper introduces a model-based reinforcement learning approach for non-Markovian systems using fractional dynamics, effectively capturing long-range dependencies in complex real-world environments.

Contribution

It proposes a novel fractional dynamical system framework for non-Markovian RL, extending traditional methods to environments with memory effects.

Findings

Fractional models effectively capture distant correlations in blood glucose data.

The approach quantifies performance differences between bounded horizon control and optimal policies.

Demonstrates applicability to real-world pharmacokinetic data.

Abstract

Reinforcement learning (RL) is a technique to learn the control policy for an agent that interacts with a stochastic environment. In any given state, the agent takes some action, and the environment determines the probability distribution over the next state as well as gives the agent some reward. Most RL algorithms typically assume that the environment satisfies Markov assumptions (i.e. the probability distribution over the next state depends only on the current state). In this paper, we propose a model-based RL technique for a system that has non-Markovian dynamics. Such environments are common in many real-world applications such as in human physiology, biological systems, material science, and population dynamics. Model-based RL (MBRL) techniques typically try to simultaneously learn a model of the environment from the data, as well as try to identify an optimal policy for the learned model. We propose a technique where the non-Markovianity of the system is modeled through a fractional dynamical system. We show that we can quantify the difference in the performance of an MBRL algorithm that uses bounded horizon model predictive control from the optimal policy. Finally, we demonstrate our proposed framework on a pharmacokinetic model of human blood glucose dynamics and show that our fractional models can capture distant correlations on real-world datasets.