# Reinforcement Learning with a Corrupted Reward Channel

## TL;DR

This paper addresses the challenge of reward corruption in reinforcement learning caused by sensory errors, proposing methods to mitigate its effects through richer data and randomisation techniques.

## Contribution

It formalizes the Corrupt Reward MDP framework and explores two strategies—richer data and randomisation—to improve RL robustness against reward corruption.

## Key findings

- Richer data can sometimes fully mitigate reward corruption.
- Randomisation can partially reduce the impact of reward corruption.
- Traditional RL methods perform poorly under reward corruption.

## Abstract

No real-world reward function is perfect. Sensory errors and software bugs may result in RL agents observing higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formalise this problem as a generalised Markov Decision Problem called Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards. Two ways around the problem are investigated. First, by giving the agent richer data, such as in inverse reinforcement learning and semi-supervised reinforcement learning, reward corruption stemming from systematic sensory errors may sometimes be completely managed. Second, by using randomisation to blunt the agent's optimisation, reward corruption can be partially managed under some assumptions.