DADAgger: Disagreement-Augmented Dataset Aggregation

TL;DR

DADAgger improves imitation learning by selectively querying experts for out-of-distribution states, reducing sample queries while maintaining performance, and enabling efficient dataset creation.

Contribution

It introduces a novel OOD detection method for selective expert querying in imitation learning, reducing sample complexity compared to DAgger.

Findings

Achieves comparable performance to DAgger with fewer expert queries

Outperforms random sampling baseline in environment tests

Can build balanced datasets with minimal initial data

Abstract

DAgger is an imitation algorithm that aggregates its original datasets by querying the expert on all samples encountered during training. In order to reduce the number of samples queried, we propose a modification to DAgger, known as DADAgger, which only queries the expert for state-action pairs that are out of distribution (OOD). OOD states are identified by measuring the variance of the action predictions of an ensemble of models on each state, which we simulate using dropout. Testing on the Car Racing and Half Cheetah environments achieves comparable performance to DAgger but with reduced expert queries, and better performance than a random sampling baseline. We also show that our algorithm may be used to build efficient, well-balanced training datasets by running with no initial data and only querying the expert to resolve uncertainty.