Evolution of Rewards for Food and Motor Action by Simulating Birth and Death

TL;DR

This study simulates the evolution of reward functions in agents, revealing how environmental factors influence reward shaping for food and motor actions, and demonstrating the emergence of biologically plausible reward patterns.

Contribution

It introduces a novel evolutionary simulation framework that models reward function evolution in agents, highlighting the impact of environment on reward development.

Findings

Positive rewards for food acquisition evolve naturally.

Motor action rewards diverge into positive and negative modes.

Reward stability varies with environmental food quality.

Abstract

The reward system is one of the fundamental drivers of animal behaviors and is critical for survival and reproduction. Despite its importance, the problem of how the reward system has evolved is underexplored. In this paper, we try to replicate the evolution of biologically plausible reward functions and investigate how environmental conditions affect evolved rewards' shape. For this purpose, we developed a population-based decentralized evolutionary simulation framework, where agents maintain their energy level to live longer and produce more children. Each agent inherits its reward function from its parent subject to mutation and learns to get rewards via reinforcement learning throughout its lifetime. Our results show that biologically reasonable positive rewards for food acquisition and negative rewards for motor action can evolve from randomly initialized ones. However, we also find that the rewards for motor action diverge into two modes: largely positive and slightly negative. The emergence of positive motor action rewards is surprising because it can make agents too active and inefficient in foraging. In environments with poor and poisonous foods, the evolution of rewards for less important foods tends to be unstable, while rewards for normal foods are still stable. These results demonstrate the usefulness of our simulation environment and energy-dependent birth and death model for further studies of the origin of reward systems.