Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?

TL;DR

This paper empirically investigates replacing value function regression with classification in offline RL, revealing mixed impacts on performance across various algorithms and tasks, highlighting potential benefits and risks.

Contribution

It provides a large-scale empirical analysis of classification-based value estimation in offline RL, offering insights into when it improves or harms performance.

Findings

Classification can outperform regression in some tasks and algorithms.

Inconsistent effects observed across different algorithms and domains.

Highlights the importance of task and algorithm considerations when using classification for value functions.

Abstract

In deep Reinforcement Learning (RL), value functions are typically approximated using deep neural networks and trained via mean squared error regression objectives to fit the true value functions. Recent research has proposed an alternative approach, utilizing the cross-entropy classification objective, which has demonstrated improved performance and scalability of RL algorithms. However, existing study have not extensively benchmarked the effects of this replacement across various domains, as the primary objective was to demonstrate the efficacy of the concept across a broad spectrum of tasks, without delving into in-depth analysis. Our work seeks to empirically investigate the impact of such a replacement in an offline RL setup and analyze the effects of different aspects on performance. Through large-scale experiments conducted across a diverse range of tasks using different algorithms, we aim to gain deeper insights into the implications of this approach. Our results reveal that incorporating this change can lead to superior performance over state-of-the-art solutions for some algorithms in certain tasks, while maintaining comparable performance levels in other tasks, however for other algorithms this modification might lead to the dramatic performance drop. This findings are crucial for further application of classification approach in research and practical tasks.