How to Provably Improve Return Conditioned Supervised Learning?

TL;DR

This paper introduces Reinforced RCSL, a framework that enhances return-conditioned supervised learning by leveraging in-distribution return-to-go, leading to provable and empirical performance improvements in offline decision-making tasks.

Contribution

It proposes a novel Reinforced RCSL framework that addresses RCSL's limitations by using in-distribution return-to-go, backed by theoretical analysis and empirical validation.

Findings

Reinforced RCSL outperforms standard RCSL in benchmarks.

Theoretical analysis confirms performance guarantees.

Empirical results show significant improvements.

Abstract

In sequential decision-making problems, Return-Conditioned Supervised Learning (RCSL) has gained increasing recognition for its simplicity and stability in modern decision-making tasks. Unlike traditional offline reinforcement learning (RL) algorithms, RCSL frames policy learning as a supervised learning problem by taking both the state and return as input. This approach eliminates the instability often associated with temporal difference (TD) learning in offline RL. However, RCSL has been criticized for lacking the stitching property, meaning its performance is inherently limited by the quality of the policy used to generate the offline dataset. To address this limitation, we propose a principled and simple framework called Reinforced RCSL. The key innovation of our framework is the introduction of a concept we call the in-distribution optimal return-to-go. This mechanism leverages our policy to identify the best achievable in-dataset future return based on the current state, avoiding the need for complex return augmentation techniques. Our theoretical analysis demonstrates that Reinforced RCSL can consistently outperform the standard RCSL approach. Empirical results further validate our claims, showing significant performance improvements across a range of benchmarks.