# Active Deep Q-learning with Demonstration

## TL;DR

This paper introduces Active Deep Q-learning with Demonstration (ARLD), enabling RL agents to actively query for expert demonstrations during training, reducing demonstration efforts and improving learning efficiency.

## Contribution

It proposes a novel active RL framework with a dynamic query strategy based on uncertainty estimation, enhancing demonstration efficiency and learning speed.

## Key findings

- Faster learning compared to passive demonstration methods.
- Achieves super-expert performance across multiple tasks.
- Uncertainty-based query strategies improve demonstration relevance.

## Abstract

Recent research has shown that although Reinforcement Learning (RL) can benefit from expert demonstration, it usually takes considerable efforts to obtain enough demonstration. The efforts prevent training decent RL agents with expert demonstration in practice. In this work, we propose Active Reinforcement Learning with Demonstration (ARLD), a new framework to streamline RL in terms of demonstration efforts by allowing the RL agent to query for demonstration actively during training. Under the framework, we propose Active Deep Q-Network, a novel query strategy which adapts to the dynamically-changing distributions during the RL training process by estimating the uncertainty of recent states. The expert demonstration data within Active DQN are then utilized by optimizing supervised max-margin loss in addition to temporal difference loss within usual DQN training. We propose two methods of estimating the uncertainty based on two state-of-the-art DQN models, namely the divergence of bootstrapped DQN and the variance of noisy DQN. The empirical results validate that both methods not only learn faster than other passive expert demonstration methods with the same amount of demonstration and but also reach super-expert level of performance across four different tasks.