Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation

TL;DR

This paper investigates how different optimization formulations in evolutionary reinforcement learning affect the diversity and effectiveness of generated behaviors, highlighting the importance of balanced multi-objective optimization.

Contribution

It analyzes the impact of reward signal modulation and evolutionary mechanisms on policy diversity within an existing framework, emphasizing the need for balanced objectives.

Findings

Unequal objective consideration reduces behavioral diversity.

Balanced multi-objective optimization improves policy effectiveness.

Unbalanced formulations can worsen task performance.

Abstract

Generating various strategies for a given task is challenging. However, it has already proven to bring many assets to the main learning process, such as improved behavior exploration. With the growth in the interest of heterogeneity in solution in evolutionary computation and reinforcement learning, many promising approaches have emerged. To better understand how one guides multiple policies toward distinct strategies and benefit from diversity, we need to analyze further the influence of the reward signal modulation and other evolutionary mechanisms on the obtained behaviors. To that effect, this paper considers an existing evolutionary reinforcement learning framework which exploits multi-objective optimization as a way to obtain policies that succeed at behavior-related tasks as well as completing the main goal. Experiments on the Atari games stress that optimization formulations which do not consider objectives equally fail at generating diversity and even output agents that are worse at solving the problem at hand, regardless of the obtained behaviors.