Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators

TL;DR

This paper introduces Temporal Reward Decomposition (TRD), a method that predicts an agent's upcoming individual rewards, providing clearer explanations of future expectations and decision-making in reinforcement learning agents.

Contribution

The paper proposes TRD, a novel approach to decompose future reward estimations into individual expected rewards, enhancing interpretability and enabling new insights into agent behavior.

Findings

TRD accurately predicts individual future rewards.

TRD enables estimation of reward timing, value, and confidence.

TRD can be integrated into DQN agents with minimal performance loss.

Abstract

Future reward estimation is a core component of reinforcement learning agents; i.e., Q-value and state-value functions, predicting an agent's sum of future rewards. Their scalar output, however, obfuscates when or what individual future rewards an agent may expect to receive. We address this by modifying an agent's future reward estimator to predict their next N expected rewards, referred to as Temporal Reward Decomposition (TRD). This unlocks novel explanations of agent behaviour. Through TRD we can: estimate when an agent may expect to receive a reward, the value of the reward and the agent's confidence in receiving it; measure an input feature's temporal importance to the agent's action decisions; and predict the influence of different actions on future rewards. Furthermore, we show that DQN agents trained on Atari environments can be efficiently retrained to incorporate TRD with minimal impact on performance.