An Error Bound for Aggregation in Approximate Dynamic Programming

TL;DR

This paper derives a broad error bound for aggregation methods in approximate dynamic programming, applicable to various aggregation schemes, aiding the analysis of RL algorithms.

Contribution

It extends a known error bound to more general aggregation schemes in DP, including soft and feature-based aggregation.

Findings

Derived a general error bound for aggregation in DP.

Bound applies to soft and feature-based aggregation schemes.

Extends previous bounds from hard aggregation to broader cases.

Abstract

We consider a general aggregation framework for discounted finite-state infinite horizon dynamic programming (DP) problems. It defines an aggregate problem whose optimal cost function can be obtained off-line by exact DP and then used as a terminal cost approximation for an on-line reinforcement learning (RL) scheme. We derive a bound on the error between the optimal cost functions of the aggregate problem and the original problem. This bound was first derived by Tsitsiklis and van Roy [TvR96] for the special case of hard aggregation. Our bound is similar but applies far more broadly, including to soft aggregation and feature-based aggregation schemes.