Towards Sample-efficient Apprenticeship Learning from Suboptimal Demonstration

TL;DR

This paper introduces S3RR, a systematic approach to improve reward learning from suboptimal demonstrations in robotics, outperforming previous noise-injection methods by better representing suboptimality.

Contribution

The paper proposes S3RR, a novel systematic method for trajectory degradation that enhances reward inference from suboptimal demonstrations, surpassing noise-injection techniques.

Findings

S3RR achieves comparable or better reward correlation than SSRR.

Systematic trajectory degradation improves reward learning accuracy.

Empirical results demonstrate S3RR's effectiveness in real-world tasks.

Abstract

Learning from Demonstration (LfD) seeks to democratize robotics by enabling non-roboticist end-users to teach robots to perform novel tasks by providing demonstrations. However, as demonstrators are typically non-experts, modern LfD techniques are unable to produce policies much better than the suboptimal demonstration. A previously-proposed framework, SSRR, has shown success in learning from suboptimal demonstration but relies on noise-injected trajectories to infer an idealized reward function. A random approach such as noise-injection to generate trajectories has two key drawbacks: 1) Performance degradation could be random depending on whether the noise is applied to vital states and 2) Noise-injection generated trajectories may have limited suboptimality and therefore will not accurately represent the whole scope of suboptimality. We present Systematic Self-Supervised Reward Regression, S3RR, to investigate systematic alternatives for trajectory degradation. We carry out empirical evaluations and find S3RR can learn comparable or better reward correlation with ground-truth against a state-of-the-art learning from suboptimal demonstration framework.