Strategy Discovery and Mixture in Lifelong Learning from Heterogeneous Demonstration

TL;DR

This paper introduces DMSRD, a novel lifelong learning algorithm that distills shared knowledge from heterogeneous demonstrations, enabling flexible, efficient, and scalable robot adaptation to personalized strategies.

Contribution

It presents a new algorithm for lifelong learning from heterogeneous demonstrations, improving policy performance and strategy understanding in robotic tasks.

Findings

77% improvement in policy returns

42% improvement in log likelihood

Stronger task reward correlation

Abstract

Learning from Demonstration (LfD) approaches empower end-users to teach robots novel tasks via demonstrations of the desired behaviors, democratizing access to robotics. A key challenge in LfD research is that users tend to provide heterogeneous demonstrations for the same task due to various strategies and preferences. Therefore, it is essential to develop LfD algorithms that ensure \textit{flexibility} (the robot adapts to personalized strategies), \textit{efficiency} (the robot achieves sample-efficient adaptation), and \textit{scalability} (robot reuses a concise set of strategies to represent a large amount of behaviors). In this paper, we propose a novel algorithm, Dynamic Multi-Strategy Reward Distillation (DMSRD), which distills common knowledge between heterogeneous demonstrations, leverages learned strategies to construct mixture policies, and continues to improve by learning from all available data. Our personalized, federated, and lifelong LfD architecture surpasses benchmarks in two continuous control problems with an average 77\% improvement in policy returns and 42\% improvement in log likelihood, alongside stronger task reward correlation and more precise strategy rewards.