Incremental Model-based Learners With Formal Learning-Time Guarantees

TL;DR

This paper introduces faster, computationally efficient model-based reinforcement learning algorithms, RTDP-RMAX and RTDP-IE, with formal PAC guarantees, balancing experience and computation in large-scale MDPs.

Contribution

It develops a general framework and algorithms that significantly reduce computational costs while maintaining PAC efficiency guarantees.

Findings

RTDP-RMAX and RTDP-IE are faster than traditional algorithms.

The algorithms retain PAC efficiency guarantees.

Experimental results show a tradeoff between computation and experience demands.

Abstract

Model-based learning algorithms have been shown to use experience efficiently when learning to solve Markov Decision Processes (MDPs) with finite state and action spaces. However, their high computational cost due to repeatedly solving an internal model inhibits their use in large-scale problems. We propose a method based on real-time dynamic programming (RTDP) to speed up two model-based algorithms, RMAX and MBIE (model-based interval estimation), resulting in computationally much faster algorithms with little loss compared to existing bounds. Specifically, our two new learning algorithms, RTDP-RMAX and RTDP-IE, have considerably smaller computational demands than RMAX and MBIE. We develop a general theoretical framework that allows us to prove that both are efficient learners in a PAC (probably approximately correct) sense. We also present an experimental evaluation of these new algorithms that helps quantify the tradeoff between computational and experience demands.