Policy Improvement via Imitation of Multiple Oracles

TL;DR

This paper introduces MAMBA, a novel imitation learning algorithm that effectively leverages multiple suboptimal oracles by using a state-wise maximum value baseline, enabling faster and more robust policy learning.

Contribution

It proposes a new baseline for multi-oracle imitation learning and develops MAMBA, a policy optimization algorithm with theoretical guarantees to outperform individual oracles.

Findings

MAMBA outperforms existing IL methods in speed and policy quality.

The state-wise maximum oracle value serves as an effective benchmark.

MAMBA can leverage multiple weak oracles to improve learning efficiency.

Abstract

Despite its promise, reinforcement learning's real-world adoption has been hampered by the need for costly exploration to learn a good policy. Imitation learning (IL) mitigates this shortcoming by using an oracle policy during training as a bootstrap to accelerate the learning process. However, in many practical situations, the learner has access to multiple suboptimal oracles, which may provide conflicting advice in a state. The existing IL literature provides a limited treatment of such scenarios. Whereas in the single-oracle case, the return of the oracle's policy provides an obvious benchmark for the learner to compete against, neither such a benchmark nor principled ways of outperforming it are known for the multi-oracle setting. In this paper, we propose the state-wise maximum of the oracle policies' values as a natural baseline to resolve conflicting advice from multiple oracles. Using a reduction of policy optimization to online learning, we introduce a novel IL algorithm MAMBA, which can provably learn a policy competitive with this benchmark. In particular, MAMBA optimizes policies by using a gradient estimator in the style of generalized advantage estimation (GAE). Our theoretical analysis shows that this design makes MAMBA robust and enables it to outperform the oracle policies by a larger margin than the IL state of the art, even in the single-oracle case. In an evaluation against standard policy gradient with GAE and AggreVaTe(D), we showcase MAMBA's ability to leverage demonstrations both from a single and from multiple weak oracles, and significantly speed up policy optimization.