Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks

TL;DR

This paper proposes Chebyshev-DQN, a novel neural network architecture that incorporates Chebyshev polynomial bases to improve function approximation in Deep Q-Networks, leading to better performance on benchmark tasks.

Contribution

Introduction of Chebyshev-DQN, integrating polynomial bases into DQN to enhance approximation capabilities and improve learning efficiency in reinforcement learning.

Findings

Chebyshev-DQN outperforms standard DQN by approximately 39% on CartPole-v1.

Moderate polynomial degree (N=4) yields optimal performance.

High polynomial degree (N=8) can negatively impact learning.

Abstract

The performance of Deep Q-Networks (DQN) is critically dependent on the ability of its underlying neural network to accurately approximate the action-value function. Standard function approximators, such as multi-layer perceptrons, may struggle to efficiently represent the complex value landscapes inherent in many reinforcement learning problems. This paper introduces a novel architecture, the Chebyshev-DQN (Ch-DQN), which integrates a Chebyshev polynomial basis into the DQN framework to create a more effective feature representation. By leveraging the powerful function approximation properties of Chebyshev polynomials, we hypothesize that the Ch-DQN can learn more efficiently and achieve higher performance. We evaluate our proposed model on the CartPole-v1 benchmark and compare it against a standard DQN with a comparable number of parameters. Our results demonstrate that the Ch-DQN with a moderate polynomial degree (N=4) achieves significantly better asymptotic performance, outperforming the baseline by approximately 39\%. However, we also find that the choice of polynomial degree is a critical hyperparameter, as a high degree (N=8) can be detrimental to learning. This work validates the potential of using orthogonal polynomial bases in deep reinforcement learning while also highlighting the trade-offs involved in model complexity.