InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problem

TL;DR

This paper introduces InferNet, a neural network-based method that infers immediate rewards from delayed rewards, improving reinforcement learning performance in noisy and delayed reward scenarios across various tasks.

Contribution

InferNet is a novel neural network algorithm that explicitly learns to infer immediate rewards from delayed rewards, addressing the temporal credit assignment problem in RL.

Findings

InferNet improves RL performance in noisy reward environments.

InferNet is effective in both online and offline RL tasks.

InferNet demonstrates robustness across simulated and real-world RL problems.

Abstract

The temporal Credit Assignment Problem (CAP) is a well-known and challenging task in AI. While Reinforcement Learning (RL), especially Deep RL, works well when immediate rewards are available, it can fail when only delayed rewards are available or when the reward function is noisy. In this work, we propose delegating the CAP to a Neural Network-based algorithm named InferNet that explicitly learns to infer the immediate rewards from the delayed rewards. The effectiveness of InferNet was evaluated on two online RL tasks: a simple GridWorld and 40 Atari games; and two offline RL tasks: GridWorld and a real-life Sepsis treatment task. For all tasks, the effectiveness of using the InferNet inferred rewards is compared against the immediate and the delayed rewards with two settings: with noisy rewards and without noise. Overall, our results show that the effectiveness of InferNet is robust against noisy reward functions and is an effective add-on mechanism for solving temporal CAP in a wide range of RL tasks, from classic RL simulation environments to a real-world RL problem and for both online and offline learning.