A Hybrid PAC Reinforcement Learning Algorithm

TL;DR

This paper introduces the Dyna-Delayed Q-learning algorithm, a hybrid PAC reinforcement learning method for MDPs that combines model-free and model-based approaches, demonstrating improved sample efficiency through theoretical analysis and numerical results.

Contribution

It presents a novel hybrid PAC RL algorithm, Dyna-Delayed Q-learning, with theoretical PAC guarantees and superior empirical sample efficiency over existing methods.

Findings

Dyna-Delayed Q-learning outperforms traditional algorithms in sample efficiency.

The PAC analysis provides theoretical guarantees for the new algorithm.

Numerical experiments confirm improved performance in practical scenarios.

Abstract

This paper offers a new hybrid probably approximately correct (PAC) reinforcement learning (RL) algorithm for Markov decision processes (MDPs) that intelligently maintains favorable features of its parents. The designed algorithm, referred to as the Dyna-Delayed Q-learning (DDQ) algorithm, combines model-free and model-based learning approaches while outperforming both in most cases. The paper includes a PAC analysis of the DDQ algorithm and a derivation of its sample complexity. Numerical results are provided to support the claim regarding the new algorithm's sample efficiency compared to its parents as well as the best known model-free and model-based algorithms in application.