Temporal Video-Language Alignment Network for Reward Shaping in Reinforcement Learning

TL;DR

This paper introduces a natural language-based reward shaping method for reinforcement learning in complex environments, improving task completion rates by mapping trajectories to language instructions and generating intermediate rewards.

Contribution

It extends the LEARN framework to map trajectories to natural language, enabling more effective reward shaping in RL for Atari games.

Findings

Ext-LEARN outperforms previous reward shaping methods.

Improves task success rates in Montezuma's Revenge.

Effective integration of language instructions enhances RL learning.

Abstract

Designing appropriate reward functions for Reinforcement Learning (RL) approaches has been a significant problem, especially for complex environments such as Atari games. Utilizing natural language instructions to provide intermediate rewards to RL agents in a process known as reward shaping can help the agent in reaching the goal state faster. In this work, we propose a natural language-based reward shaping approach that maps trajectories from the Montezuma's Revenge game environment to corresponding natural language instructions using an extension of the LanguagE-Action Reward Network (LEARN) framework. These trajectory-language mappings are further used to generate intermediate rewards which are integrated into reward functions that can be utilized to learn an optimal policy for any standard RL algorithms. For a set of 15 tasks from Atari's Montezuma's Revenge game, the Ext-LEARN approach leads to the successful completion of tasks more often on average than the reward shaping approach that uses the LEARN framework and performs even better than the reward shaping framework without natural language-based rewards.