Gradient Boosting Reinforcement Learning

TL;DR

Gradient Boosting Reinforcement Learning (GBRL) adapts gradient boosting trees to RL, excelling with structured data and out-of-distribution robustness, outperforming neural networks in certain domains.

Contribution

This paper introduces GBRL, a novel framework that integrates gradient boosting trees into reinforcement learning, addressing limitations of neural networks with structured and categorical data.

Findings

GBRL outperforms neural networks on structured observation domains.

GBRL maintains competitive performance on continuous control benchmarks.

GBRL shows superior robustness to out-of-distribution samples.

Abstract

We present Gradient Boosting Reinforcement Learning (GBRL), a framework that adapts the strengths of gradient boosting trees (GBT) to reinforcement learning (RL) tasks. While neural networks (NNs) have become the de facto choice for RL, they face significant challenges with structured and categorical features and tend to generalize poorly to out-of-distribution samples. These are challenges for which GBTs have traditionally excelled in supervised learning. However, GBT's application in RL has been limited. The design of traditional GBT libraries is optimized for static datasets with fixed labels, making them incompatible with RL's dynamic nature, where both state distributions and reward signals evolve during training. GBRL overcomes this limitation by continuously interleaving tree construction with environment interaction. Through extensive experiments, we demonstrate that GBRL outperforms NNs in domains with structured observations and categorical features while maintaining competitive performance on standard continuous control benchmarks. Like its supervised learning counterpart, GBRL demonstrates superior robustness to out-of-distribution samples and better handles irregular state-action relationships.