Reward Shaping Using Convolutional Neural Network

TL;DR

This paper introduces VIN-RS, a CNN-based reward shaping method that estimates environment transition matrices and improves learning efficiency in various environments.

Contribution

It presents a novel potential-based reward shaping mechanism using CNNs trained on environment data, automatically inferring transition matrices for enhanced learning.

Findings

Improves learning speed in tabular, Atari, and MuJoCo environments.

Achieves higher maximum cumulative rewards than existing methods.

Effectively estimates transition matrices from environment observations.

Abstract

In this paper, we propose Value Iteration Network for Reward Shaping (VIN-RS), a potential-based reward shaping mechanism using Convolutional Neural Network (CNN). The proposed VIN-RS embeds a CNN trained on computed labels using the message passing mechanism of the Hidden Markov Model. The CNN processes images or graphs of the environment to predict the shaping values. Recent work on reward shaping still has limitations towards training on a representation of the Markov Decision Process (MDP) and building an estimate of the transition matrix. The advantage of VIN-RS is to construct an effective potential function from an estimated MDP while automatically inferring the environment transition matrix. The proposed VIN-RS estimates the transition matrix through a self-learned convolution filter while extracting environment details from the input frames or sampled graphs. Due to (1) the previous success of using message passing for reward shaping; and (2) the CNN planning behavior, we use these messages to train the CNN of VIN-RS. Experiments are performed on tabular games, Atari 2600 and MuJoCo, for discrete and continuous action space. Our results illustrate promising improvements in the learning speed and maximum cumulative reward compared to the state-of-the-art.