Fractional Policy Gradients: Reinforcement Learning with Long-Term Memory

TL;DR

Fractional Policy Gradients introduce fractional calculus into reinforcement learning to model long-term dependencies, reducing variance and improving sample efficiency without extra computational costs.

Contribution

This paper presents a novel reinforcement learning framework using fractional derivatives to capture long-term temporal correlations in policy optimization.

Findings

Achieves 35-68% sample efficiency gains.

Reduces variance by 24-52% compared to baselines.

Theoretically guarantees asymptotic variance reduction.

Abstract

We propose Fractional Policy Gradients (FPG), a reinforcement learning framework incorporating fractional calculus for long-term temporal modeling in policy optimization. Standard policy gradient approaches face limitations from Markovian assumptions, exhibiting high variance and inefficient sampling. By reformulating gradients using Caputo fractional derivatives, FPG establishes power-law temporal correlations between state transitions. We develop an efficient recursive computation technique for fractional temporal-difference errors with constant time and memory requirements. Theoretical analysis shows FPG achieves asymptotic variance reduction of order O(t^(-alpha)) versus standard policy gradients while preserving convergence. Empirical validation demonstrates 35-68% sample efficiency gains and 24-52% variance reduction versus state-of-the-art baselines. This framework provides a mathematically grounded approach for leveraging long-range dependencies without computational overhead.