A Matrix Splitting Perspective on Planning with Options

TL;DR

This paper presents a novel perspective on the options framework in reinforcement learning by relating the Bellman operator to matrix splitting, revealing how options' timescales influence convergence rates and computational trade-offs.

Contribution

It introduces a matrix splitting viewpoint for the options framework, connecting convergence behavior to options' timescales and highlighting computational trade-offs.

Findings

Convergence rate depends on options' inherent timescales.

Trade-off identified between asymptotic performance and computational cost.

Matrix splitting perspective offers new insights into options-based planning.

Abstract

We show that the Bellman operator underlying the options framework leads to a matrix splitting, an approach traditionally used to speed up convergence of iterative solvers for large linear systems of equations. Based on standard comparison theorems for matrix splittings, we then show how the asymptotic rate of convergence varies as a function of the inherent timescales of the options. This new perspective highlights a trade-off between asymptotic performance and the cost of computation associated with building a good set of options.