Student-Initiated Action Advising via Advice Novelty

TL;DR

This paper introduces a student-initiated action advising method in deep reinforcement learning that uses Random Network Distillation to measure advice novelty, addressing limitations of existing approaches and improving learning efficiency.

Contribution

The proposed algorithm employs RND for advice novelty and updates only advised states, enhancing robustness over existing state novelty-based methods.

Findings

Performs comparably to state-of-the-art methods in standard scenarios.

Shows significant advantages in scenarios where existing methods fail.

Effectively mitigates feedback lag issues in advice timing.

Abstract

Action advising is a budget-constrained knowledge exchange mechanism between teacher-student peers that can help tackle exploration and sample inefficiency problems in deep reinforcement learning (RL). Most recently, student-initiated techniques that utilise state novelty and uncertainty estimations have obtained promising results. However, the approaches built on these estimations have some potential weaknesses. First, they assume that the convergence of the student's RL model implies less need for advice. This can be misleading in scenarios with teacher absence early on where the student is likely to learn suboptimally by itself; yet also ignore the teacher's assistance later. Secondly, the delays between encountering states and having them to take effect in the RL model updates in presence of the experience replay dynamics cause a feedback lag in what the student actually needs advice for. We propose a student-initiated algorithm that alleviates these by employing Random Network Distillation (RND) to measure the novelty of a piece of advice. Furthermore, we perform RND updates only for the advised states to ensure that the student's own learning does not impair its ability to leverage the teacher. Experiments in GridWorld and MinAtar show that our approach performs on par with the state-of-the-art and demonstrates significant advantages in the scenarios where the existing methods are prone to fail.