Moral reinforcement learning using actual causation

TL;DR

This paper introduces a reinforcement learning method that incorporates actual causation to ensure agents avoid causing harm, aligning their behavior with moral expectations in decision-making.

Contribution

It proposes an online reinforcement learning approach that uses actual causation theory to prevent agents from being the cause of harm, a novel integration of moral causality into RL.

Findings

The method successfully avoids harmful behavior in a toy ethical dilemma.

It demonstrates that causal moral distinctions can guide policy learning.

The approach aligns agent behavior with moral judgments based on causality.

Abstract

Reinforcement learning systems will to a greater and greater extent make decisions that significantly impact the well-being of humans, and it is therefore essential that these systems make decisions that conform to our expectations of morally good behavior. The morally good is often defined in causal terms, as in whether one's actions have in fact caused a particular outcome, and whether the outcome could have been anticipated. We propose an online reinforcement learning method that learns a policy under the constraint that the agent should not be the cause of harm. This is accomplished by defining cause using the theory of actual causation and assigning blame to the agent when its actions are the actual cause of an undesirable outcome. We conduct experiments on a toy ethical dilemma in which a natural choice of reward function leads to clearly undesirable behavior, but our method learns a policy that avoids being the cause of harmful behavior, demonstrating the soundness of our approach. Allowing an agent to learn while observing causal moral distinctions such as blame, opens the possibility to learning policies that better conform to our moral judgments.