Cooperative Deep $Q$-learning Framework for Environments Providing Image Feedback

TL;DR

This paper introduces a dual neural network deep Q-learning framework that enhances sample efficiency and accelerates learning in environments with image feedback by using error-driven updates and theoretical guarantees.

Contribution

It proposes a novel dual neural network approach with error-driven learning for improved sample efficiency and convergence in deep reinforcement learning with image inputs.

Findings

Faster learning and convergence demonstrated in simulations

Reduced buffer size improves sample efficiency

Theoretical proof of approximation error reduction over time

Abstract

In this paper, we address two key challenges in deep reinforcement learning setting, sample inefficiency and slow learning, with a dual NN-driven learning approach. In the proposed approach, we use two deep NNs with independent initialization to robustly approximate the action-value function in the presence of image inputs. In particular, we develop a temporal difference (TD) error-driven learning approach, where we introduce a set of linear transformations of the TD error to directly update the parameters of each layer in the deep NN. We demonstrate theoretically that the cost minimized by the error-driven learning (EDL) regime is an approximation of the empirical cost and the approximation error reduces as learning progresses, irrespective of the size of the network. Using simulation analysis, we show that the proposed methods enables faster learning and convergence and requires reduced buffer size (thereby increasing the sample efficiency).