Combining Automated Optimisation of Hyperparameters and Reward Shape

TL;DR

This paper presents a methodology for jointly optimizing hyperparameters and reward functions in deep reinforcement learning, demonstrating improved performance and stability across multiple environments.

Contribution

It introduces a combined optimization approach for hyperparameters and reward functions, highlighting their mutual dependence and enhancing RL performance.

Findings

Combined optimization outperforms baseline in half of the environments.

Including a variance penalty improves policy stability.

The approach achieves competitive results with minimal additional computational cost.

Abstract

There has been significant progress in deep reinforcement learning (RL) in recent years. Nevertheless, finding suitable hyperparameter configurations and reward functions remains challenging even for experts, and performance heavily relies on these design choices. Also, most RL research is conducted on known benchmarks where knowledge about these choices already exists. However, novel practical applications often pose complex tasks for which no prior knowledge about good hyperparameters and reward functions is available, thus necessitating their derivation from scratch. Prior work has examined automatically tuning either hyperparameters or reward functions individually. We demonstrate empirically that an RL algorithm's hyperparameter configurations and reward function are often mutually dependent, meaning neither can be fully optimised without appropriate values for the other. We then propose a methodology for the combined optimisation of hyperparameters and the reward function. Furthermore, we include a variance penalty as an optimisation objective to improve the stability of learned policies. We conducted extensive experiments using Proximal Policy Optimisation and Soft Actor-Critic on four environments. Our results show that combined optimisation significantly improves over baseline performance in half of the environments and achieves competitive performance in the others, with only a minor increase in computational costs. This suggests that combined optimisation should be best practice.