A Benchmark for Low-Switching-Cost Reinforcement Learning

TL;DR

This paper introduces the first empirical benchmark for low-switching-cost reinforcement learning, evaluating various approaches across multiple environments to balance reward maximization with minimal policy changes.

Contribution

It systematically compares different low-switching-cost RL methods and provides insights into reducing policy switches without sacrificing sample efficiency.

Findings

Certain approaches significantly reduce policy switches.

Maintained sample efficiency under low-switching constraints.

Benchmark results guide future low-switching RL development.

Abstract

A ubiquitous requirement in many practical reinforcement learning (RL) applications, including medical treatment, recommendation system, education and robotics, is that the deployed policy that actually interacts with the environment cannot change frequently. Such an RL setting is called low-switching-cost RL, i.e., achieving the highest reward while reducing the number of policy switches during training. Despite the recent trend of theoretical studies aiming to design provably efficient RL algorithms with low switching costs, none of the existing approaches have been thoroughly evaluated in popular RL testbeds. In this paper, we systematically studied a wide collection of policy-switching approaches, including theoretically guided criteria, policy-difference-based methods, and non-adaptive baselines. Through extensive experiments on a medical treatment environment, the Atari games, and robotic control tasks, we present the first empirical benchmark for low-switching-cost RL and report novel findings on how to decrease the switching cost while maintain a similar sample efficiency to the case without the low-switching-cost constraint. We hope this benchmark could serve as a starting point for developing more practically effective low-switching-cost RL algorithms. We release our code and complete results in https://sites.google.com/view/low-switching-cost-rl.