A Survey of Temporal Credit Assignment in Deep Reinforcement Learning

TL;DR

This survey reviews the state of the art in Temporal Credit Assignment in deep reinforcement learning, proposing a unifying formalism, analyzing challenges, and discussing evaluation protocols to advance understanding and research in the field.

Contribution

It introduces a unifying formalism for credit, compares algorithms, and provides a comprehensive overview of challenges and evaluation methods in Temporal Credit Assignment for deep RL.

Findings

Proposes a formalism for credit that enables algorithm comparison.

Analyzes challenges like delayed effects and lack of influence.

Suggests evaluation protocols and future research directions.

Abstract

The Credit Assignment Problem (CAP) refers to the longstanding challenge of Reinforcement Learning (RL) agents to associate actions with their long-term consequences. Solving the CAP is a crucial step towards the successful deployment of RL in the real world since most decision problems provide feedback that is noisy, delayed, and with little or no information about the causes. These conditions make it hard to distinguish serendipitous outcomes from those caused by informed decision-making. However, the mathematical nature of credit and the CAP remains poorly understood and defined. In this survey, we review the state of the art of Temporal Credit Assignment (CA) in deep RL. We propose a unifying formalism for credit that enables equitable comparisons of state-of-the-art algorithms and improves our understanding of the trade-offs between the various methods. We cast the CAP as the problem of learning the influence of an action over an outcome from a finite amount of experience. We discuss the challenges posed by delayed effects, transpositions, and a lack of action influence, and analyse how existing methods aim to address them. Finally, we survey the protocols to evaluate a credit assignment method and suggest ways to diagnose the sources of struggle for different methods. Overall, this survey provides an overview of the field for new-entry practitioners and researchers, it offers a coherent perspective for scholars looking to expedite the starting stages of a new study on the CAP, and it suggests potential directions for future research.