Temporal Difference Learning with Constrained Initial Representations

TL;DR

This paper proposes a novel framework called CIR that constrains initial representations in off-policy RL using Tanh activation, leading to improved stability and sample efficiency, with strong empirical results on control tasks.

Contribution

It introduces the CIR framework with Tanh-based constraints, skip connections, and convex Q-learning, providing theoretical analysis and superior empirical performance.

Findings

CIR outperforms baseline methods on continuous control tasks.

Theoretical analysis confirms convergence properties of the proposed approach.

Empirical results demonstrate improved stability and sample efficiency.

Abstract

Recently, there have been numerous attempts to enhance the sample efficiency of off-policy reinforcement learning (RL) agents when interacting with the environment, including architecture improvements and new algorithms. Despite these advances, they overlook the potential of directly constraining the initial representations of the input data, which can intuitively alleviate the distribution shift issue and stabilize training. In this paper, we introduce the Tanh function into the initial layer to fulfill such a constraint. We theoretically unpack the convergence property of the temporal difference learning with the Tanh function under linear function approximation. Motivated by theoretical insights, we present our Constrained Initial Representations framework, tagged CIR, which is made up of three components: (i) the Tanh activation along with normalization methods to stabilize representations; (ii) the skip connection module to provide a linear pathway from the shallow layer to the deep layer; (iii) the convex Q-learning that allows a more flexible value estimate and mitigates potential conservatism. Empirical results show that CIR exhibits strong performance on numerous continuous control tasks, even being competitive or surpassing existing strong baseline methods.