Better Safe than Sorry: Evidence Accumulation Allows for Safe Reinforcement Learning

TL;DR

This paper introduces an evidence accumulation module for reinforcement learning agents, enabling them to delay decisions until sufficiently confident, which improves safety and performance in uncertain, stochastic environments.

Contribution

It proposes a novel accumulator-based decision mechanism inspired by biological decision-making, allowing RL agents to act only when confident, reducing errors caused by premature actions.

Findings

Achieves near-optimal performance on a guessing game

Outperforms traditional deep recurrent networks

Enhances safety by delaying decisions until confidence is high

Abstract

In the real world, agents often have to operate in situations with incomplete information, limited sensing capabilities, and inherently stochastic environments, making individual observations incomplete and unreliable. Moreover, in many situations it is preferable to delay a decision rather than run the risk of making a bad decision. In such situations it is necessary to aggregate information before taking an action; however, most state of the art reinforcement learning (RL) algorithms are biased towards taking actions \textit{at every time step}, even if the agent is not particularly confident in its chosen action. This lack of caution can lead the agent to make critical mistakes, regardless of prior experience and acclimation to the environment. Motivated by theories of dynamic resolution of uncertainty during decision making in biological brains, we propose a simple accumulator module which accumulates evidence in favor of each possible decision, encodes uncertainty as a dynamic competition between actions, and acts on the environment only when it is sufficiently confident in the chosen action. The agent makes no decision by default, and the burden of proof to make a decision falls on the policy to accrue evidence strongly in favor of a single decision. Our results show that this accumulator module achieves near-optimal performance on a simple guessing game, far outperforming deep recurrent networks using traditional, forced action selection policies.