# Emotion in Reinforcement Learning Agents and Robots: A Survey

## TL;DR

This survey reviews computational emotion models in reinforcement learning agents, highlighting their roles in decision-making, learning efficiency, and social signaling, and discusses modeling approaches, types of emotions, and evaluation criteria.

## Contribution

It provides the first comprehensive overview of emotion modeling in RL agents, connecting theory, implementation, and evaluation, and suggests future research directions.

## Key findings

- Emotion models can improve RL learning efficiency.
- Emotions serve as social signals in RL agents.
- Comparison of evaluation criteria for emotion models in RL.

## Abstract

This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computational emotion models are usually grounded in the agent's decision making architecture, of which RL is an important subclass. Studying emotions in RL-based agents is useful for three research fields. For machine learning (ML) researchers, emotion models may improve learning efficiency. For the interactive ML and human-robot interaction (HRI) community, emotions can communicate state and enhance user investment. Lastly, it allows affective modelling (AM) researchers to investigate their emotion theories in a successful AI agent class. This survey provides background on emotion theory and RL. It systematically addresses 1) from what underlying dimensions (e.g., homeostasis, appraisal) emotions can be derived and how these can be modelled in RL-agents, 2) what types of emotions have been derived from these dimensions, and 3) how these emotions may either influence the learning efficiency of the agent or be useful as social signals. We also systematically compare evaluation criteria, and draw connections to important RL sub-domains like (intrinsic) motivation and model-based RL. In short, this survey provides both a practical overview for engineers wanting to implement emotions in their RL agents, and identifies challenges and directions for future emotion-RL research.