Combining imitation and deep reinforcement learning to accomplish human-level performance on a virtual foraging task

TL;DR

This paper presents a framework combining imitation learning and deep reinforcement learning to develop bio-inspired foraging policies that match human performance in a virtual environment, emphasizing the importance of combined spatial representations.

Contribution

It introduces a novel approach integrating IL and RL for bio-inspired foraging, demonstrating improved policy performance over passive imitation alone.

Findings

Passive imitation underperforms humans in foraging tasks.

RL refinement improves policy stability and performance.

Combining allocentric and egocentric information is crucial for success.

Abstract

We develop a simple framework to learn bio-inspired foraging policies using human data. We conduct an experiment where humans are virtually immersed in an open field foraging environment and are trained to collect the highest amount of rewards. A Markov Decision Process (MDP) framework is introduced to model the human decision dynamics. Then, Imitation Learning (IL) based on maximum likelihood estimation is used to train Neural Networks (NN) that map human decisions to observed states. The results show that passive imitation substantially underperforms humans. We further refine the human-inspired policies via Reinforcement Learning (RL) using the on-policy Proximal Policy Optimization (PPO) algorithm which shows better stability than other algorithms and can steadily improve the policies pretrained with IL. We show that the combination of IL and RL can match human results and that good performance strongly depends on combining the allocentric information with an egocentric representation of the environment.