Gamma-Nets: Generalizing Value Estimation over Timescale

TL;DR

Gamma-Nets enable flexible value estimation across multiple timescales, allowing predictions at arbitrary durations without prior task knowledge, demonstrated across various RL settings including Atari games.

Contribution

Introduces Gamma-Nets, a novel method for generalizing value estimation over timescales by incorporating timescale as an input, facilitating multi-timescale predictions.

Findings

Effective in policy evaluation on square wave and robot arm tasks.

Maintains high accuracy with minimal cost when predicting multiple timescales.

Applicable to deep RL environments like Atari games.

Abstract

We present $\Gamma$-nets, a method for generalizing value function estimation over timescale. By using the timescale as one of the estimator's inputs we can estimate value for arbitrary timescales. As a result, the prediction target for any timescale is available and we are free to train on multiple timescales at each timestep. Here we empirically evaluate $\Gamma$-nets in the policy evaluation setting. We first demonstrate the approach on a square wave and then on a robot arm using linear function approximation. Next, we consider the deep reinforcement learning setting using several Atari video games. Our results show that $\Gamma$-nets can be effective for predicting arbitrary timescales, with only a small cost in accuracy as compared to learning estimators for fixed timescales. $\Gamma$-nets provide a method for compactly making predictions at many timescales without requiring a priori knowledge of the task, making it a valuable contribution to ongoing work on model-based planning, representation learning, and lifelong learning algorithms.