Batch-iFDD for Representation Expansion in Large MDPs

TL;DR

This paper introduces Batch-iFDD, a scalable feature construction method for large MDPs that improves upon existing matching pursuit algorithms by reducing the need for large feature pools and ensuring convergence.

Contribution

Batch-iFDD is a new MP-based feature discovery algorithm that guarantees convergence and scales efficiently without requiring extensive feature pools.

Findings

Outperforms previous MP algorithms in large-scale domains

Demonstrates scalability with up to one million states

Achieves effective value function approximation

Abstract

Matching pursuit (MP) methods are a promising class of feature construction algorithms for value function approximation. Yet existing MP methods require creating a pool of potential features, mandating expert knowledge or enumeration of a large feature pool, both of which hinder scalability. This paper introduces batch incremental feature dependency discovery (Batch-iFDD) as an MP method that inherits a provable convergence property. Additionally, Batch-iFDD does not require a large pool of features, leading to lower computational complexity. Empirical policy evaluation results across three domains with up to one million states highlight the scalability of Batch-iFDD over the previous state of the art MP algorithm.