To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning

TL;DR

This paper introduces a novel approach to offline reinforcement learning that balances the benefits and costs of policy switching using optimal transport, with a new algorithm demonstrated on robotics and traffic control benchmarks.

Contribution

It formulates the problem of policy switching in offline RL with a principled approach based on optimal transport and develops a Net Actor-Critic algorithm for this task.

Findings

Efficient policy switching demonstrated on robot control benchmarks.

Effective handling of switching costs in offline RL scenarios.

Improved performance in traffic light control tasks.

Abstract

Reinforcement learning (RL) -- finding the optimal behaviour (also referred to as policy) maximizing the collected long-term cumulative reward -- is among the most influential approaches in machine learning with a large number of successful applications. In several decision problems, however, one faces the possibility of policy switching -- changing from the current policy to a new one -- which incurs a non-negligible cost, and in the decision one is limited to using historical data without the availability for further online interaction. Despite the inevitable importance of this offline learning scenario, to our best knowledge, very little effort has been made to tackle the key problem of balancing between the gain and the cost of switching in a flexible and principled way. Leveraging ideas from the area of optimal transport, we initialize the systematic study of policy switching in offline RL. We establish fundamental properties and design a Net Actor-Critic algorithm for the proposed novel switching formulation. Numerical experiments demonstrate the efficiency of our approach on multiple robot control benchmarks of the Gymnasium and traffic light control from SUMO-RL.