Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

TL;DR

This large-scale benchmarking study evaluates how user experience, task complexity, and start configuration influence the performance of various motion-based learning from demonstration methods on a physical robot.

Contribution

The paper provides a comprehensive empirical comparison of four learning from demonstration approaches across diverse tasks and conditions, with publicly available dataset and practical guidelines.

Findings

Task complexity significantly affects performance.

User expertise influences learning success.

Start configuration impacts task reproduction accuracy.

Abstract

In this work, we contribute a large-scale study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these learning techniques. In particular, we evaluate four different approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four different manipulation tasks with varying starting conditions. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are being made publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.