SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations

TL;DR

SPLASH introduces a sample-efficient preference-based IRL method capable of learning long-horizon, adversarial tasks from suboptimal hierarchical demonstrations, outperforming existing approaches in simulation and real-world maritime scenarios.

Contribution

The paper presents SPLASH, a novel IRL framework that effectively learns from suboptimal, hierarchical demonstrations for complex long-horizon and adversarial tasks.

Findings

SPLASH outperforms state-of-the-art reward learning methods.

Effective in simulation for maritime capture-the-flag.

Demonstrates real-world applicability with autonomous surface vehicles.

Abstract

Inverse Reinforcement Learning (IRL) presents a powerful paradigm for learning complex robotic tasks from human demonstrations. However, most approaches make the assumption that expert demonstrations are available, which is often not the case. Those that allow for suboptimality in the demonstrations are not designed for long-horizon goals or adversarial tasks. Many desirable robot capabilities fall into one or both of these categories, thus highlighting a critical shortcoming in the ability of IRL to produce field-ready robotic agents. We introduce Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations (SPLASH), which advances the state-of-the-art in learning from suboptimal demonstrations to long-horizon and adversarial settings. We empirically validate SPLASH on a maritime capture-the-flag task in simulation, and demonstrate real-world applicability with sim-to-real translation experiments on autonomous unmanned surface vehicles. We show that our proposed methods allow SPLASH to significantly outperform the state-of-the-art in reward learning from suboptimal demonstrations.