Innate-Values-driven Reinforcement Learning based Cognitive Modeling

TL;DR

This paper introduces a novel reinforcement learning model driven by innate values, enabling agents to better balance internal needs and external rewards, leading to improved performance in complex tasks.

Contribution

It proposes the innate-values-driven RL (IVRL) framework and two models, IV-DQN and IV-A2C, integrating intrinsic motivations into RL for more human-like decision-making.

Findings

IVRL models outperform benchmark algorithms in VIZDoom RPG tasks.

IVRL enables better internal needs management and goal organization.

Models demonstrate improved learning efficiency and adaptability.

Abstract

Innate values describe agents' intrinsic motivations, which reflect their inherent interests and preferences for pursuing goals and drive them to develop diverse skills that satisfy their various needs. Traditional reinforcement learning (RL) is learning from interaction based on the feedback rewards of the environment. However, in real scenarios, the rewards are generated by agents' innate value systems, which differ vastly from individuals based on their needs and requirements. In other words, considering the AI agent as a self-organizing system, developing its awareness through balancing internal and external utilities based on its needs in different tasks is a crucial problem for individuals learning to support others and integrate community with safety and harmony in the long term. To address this gap, we propose a new RL model termed innate-values-driven RL (IVRL) based on combined motivations' models and expected utility theory to mimic its complex behaviors in the evolution through decision-making and learning. Then, we introduce two IVRL-based models: IV-DQN and IV-A2C. By comparing them with benchmark algorithms such as DQN, DDQN, A2C, and PPO in the Role-Playing Game (RPG) reinforcement learning test platform VIZDoom, we demonstrated that the IVRL-based models can help the agent rationally organize various needs, achieve better performance effectively.