Learning to Win, Lose and Cooperate through Reward Signal Evolution

TL;DR

This paper presents an evolutionary method to automatically discover reward signals enabling reinforcement learning agents to learn complex behaviors like winning, losing, and cooperating in Pong without predefined reward functions.

Contribution

It introduces a framework for optimizing multiple goals with evolved reward signals, demonstrating successful learning of high-level behaviors from scratch.

Findings

Agents learned high-level goals without prespecified rewards

Evolved reward signals led to surprising and effective behaviors

Training stability improved with the evolutionary reward approach

Abstract

Solving a reinforcement learning problem typically involves correctly prespecifying the reward signal from which the algorithm learns. Here, we approach the problem of reward signal design by using an evolutionary approach to perform a search on the space of all possible reward signals. We introduce a general framework for optimizing $N$ goals given $n$ reward signals. Through experiments we demonstrate that such an approach allows agents to learn high-level goals - such as winning, losing and cooperating - from scratch without prespecified reward signals in the game of Pong. Some of the solutions found by the algorithm are surprising, in the sense that they would probably not have been chosen by a person trying to hand-code a given behaviour through a specific reward signal. Furthermore, it seems that the proposed approach may also benefit from higher stability of the training performance when compared with the typical score-based reward signals.