RadGrad: Active learning with loss gradients

TL;DR

RadGrad is an active learning algorithm for sequential decision problems that reduces expert queries by predicting agent error and risk, maintaining performance while lowering costs, but faces challenges in complex environments.

Contribution

Introduces RadGrad, an active learning method that selectively queries experts based on error and risk predictions, improving efficiency over existing algorithms.

Findings

RadGrad matches or exceeds DAgger in simple environments.

Complex environments challenge RadGrad and other safety-aware algorithms.

RadGrad reduces expert queries significantly while maintaining performance.

Abstract

Solving sequential decision prediction problems, including those in imitation learning settings, requires mitigating the problem of covariate shift. The standard approach, DAgger, relies on capturing expert behaviour in all states that the agent reaches. In real-world settings, querying an expert is costly. We propose a new active learning algorithm that selectively queries the expert, based on both a prediction of agent error and a proxy for agent risk, that maintains the performance of unrestrained expert querying systems while substantially reducing the number of expert queries made. We show that our approach, RadGrad, has the potential to improve upon existing safety-aware algorithms, and matches or exceeds the performance of DAgger and variants (i.e., SafeDAgger) in one simulated environment. However, we also find that a more complex environment poses challenges not only to our proposed method, but also to existing safety-aware algorithms, which do not match the performance of DAgger in our experiments.