Learning on a Budget via Teacher Imitation

TL;DR

This paper introduces a unified, adaptive approach to learning from teacher advice in reinforcement learning, optimizing advice collection and utilization under budget constraints, and demonstrating strong performance in Atari games.

Contribution

It extends teacher imitation to unify advice collection and use, with automatic hyperparameter tuning for broad applicability and simplicity.

Findings

Outperforms or matches top competitors in Atari games

Components provide significant individual advantages

Automatically adapts to different tasks with minimal human intervention

Abstract

Deep Reinforcement Learning (RL) techniques can benefit greatly from leveraging prior experience, which can be either self-generated or acquired from other entities. Action advising is a framework that provides a flexible way to transfer such knowledge in the form of actions between teacher-student peers. However, due to the realistic concerns, the number of these interactions is limited with a budget; therefore, it is crucial to perform these in the most appropriate moments. There have been several promising studies recently that address this problem setting especially from the student's perspective. Despite their success, they have some shortcomings when it comes to the practical applicability and integrity as an overall solution to the learning from advice challenge. In this paper, we extend the idea of advice reusing via teacher imitation to construct a unified approach that addresses both advice collection and advice utilisation problems. We also propose a method to automatically tune the relevant hyperparameters of these components on-the-fly to make it able to adapt to any task with minimal human intervention. The experiments we performed in 5 different Atari games verify that our algorithm either surpasses or performs on-par with its top competitors while being far simpler to be employed. Furthermore, its individual components are also found to be providing significant advantages alone.