# Source Traces for Temporal Difference Learning

## TL;DR

This paper introduces source traces for TD learning, which model potential histories to improve propagation of errors and generalization, offering new algorithms and insights into combining with existing methods.

## Contribution

It develops the concept of source traces, proposes a TD($$)-like learning algorithm, and introduces a new method for learning source maps that outperform previous algorithms.

## Key findings

- Source traces enable faster error propagation and generalization.
- The proposed algorithms converge and outperform previous methods.
- Source traces can be effectively integrated with other model-based approaches.

## Abstract

This paper motivates and develops source traces for temporal difference (TD) learning in the tabular setting. Source traces are like eligibility traces, but model potential histories rather than immediate ones. This allows TD errors to be propagated to potential causal states and leads to faster generalization. Source traces can be thought of as the model-based, backward view of successor representations (SR), and share many of the same benefits. This view, however, suggests several new ideas. First, a TD($\lambda$)-like source learning algorithm is proposed and its convergence is proven. Then, a novel algorithm for learning the source map (or SR matrix) is developed and shown to outperform the previous algorithm. Finally, various approaches to using the source/SR model are explored, and it is shown that source traces can be effectively combined with other model-based methods like Dyna and experience replay.