Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations

TL;DR

This paper assesses the robustness of offline reinforcement learning methods for robot control, revealing significant vulnerabilities to action perturbations and emphasizing the need for more resilient algorithms.

Contribution

It provides the first systematic evaluation of offline reinforcement learning robustness against action perturbations in robotic systems.

Findings

Offline RL methods are more vulnerable than online methods to action perturbations.

Existing offline RL approaches show significant performance drops under simulated failures.

Robustness evaluation highlights critical challenges for deploying offline RL in real-world robots.

Abstract

Offline reinforcement learning, which learns solely from datasets without environmental interaction, has gained attention. This approach, similar to traditional online deep reinforcement learning, is particularly promising for robot control applications. Nevertheless, its robustness against real-world challenges, such as joint actuator faults in robots, remains a critical concern. This study evaluates the robustness of existing offline reinforcement learning methods using legged robots from OpenAI Gym based on average episodic rewards. For robustness evaluation, we simulate failures by incorporating both random and adversarial perturbations, representing worst-case scenarios, into the joint torque signals. Our experiments show that existing offline reinforcement learning methods exhibit significant vulnerabilities to these action perturbations and are more vulnerable than online reinforcement learning methods, highlighting the need for more robust approaches in this field.