Appraisal-Guided Proximal Policy Optimization: Modeling Psychological Disorders in Dynamic Grid World

TL;DR

This paper introduces an appraisal-guided reinforcement learning method to model and simulate psychological disorders like anxiety and OCD in agents within a dynamic environment, improving generalization and behavioral analysis.

Contribution

It develops a novel AG-PPO algorithm incorporating appraisal theory and reward shaping to simulate psychological disorders in RL agents, enhancing behavioral modeling and analysis.

Findings

AG-PPO outperforms standard PPO in generalization.

Different reward strategies successfully simulate anxiety and OCD behaviors.

Behavioral patterns align with psychological disorder characteristics.

Abstract

The integration of artificial intelligence across multiple domains has emphasized the importance of replicating human-like cognitive processes in AI. By incorporating emotional intelligence into AI agents, their emotional stability can be evaluated to enhance their resilience and dependability in critical decision-making tasks. In this work, we develop a methodology for modeling psychological disorders using Reinforcement Learning (RL) agents. We utilized Appraisal theory to train RL agents in a dynamic grid world environment with an Appraisal-Guided Proximal Policy Optimization (AG-PPO) algorithm. Additionally, we investigated numerous reward-shaping strategies to simulate psychological disorders and regulate the behavior of the agents. A comparison of various configurations of the modified PPO algorithm identified variants that simulate Anxiety disorder and Obsessive-Compulsive Disorder (OCD)-like behavior in agents. Furthermore, we compared standard PPO with AG-PPO and its configurations, highlighting the performance improvement in terms of generalization capabilities. Finally, we conducted an analysis of the agents' behavioral patterns in complex test environments to evaluate the associated symptoms corresponding to the psychological disorders. Overall, our work showcases the benefits of the appraisal-guided PPO algorithm over the standard PPO algorithm and the potential to simulate psychological disorders in a controlled artificial environment and evaluate them on RL agents.