Back to Reality for Imitation Learning

TL;DR

This paper argues that imitation learning evaluation should prioritize time efficiency over data efficiency to better reflect real-world robotics costs, urging the community to develop new, task-specific metrics.

Contribution

It highlights the need for new evaluation metrics in robot learning that focus on time efficiency, aligning better with real-world application costs.

Findings

Current metrics focus on data efficiency, not real-world costs.

Time efficiency is a more relevant metric for practical robot learning.

Call to develop new evaluation standards tailored for robotics.

Abstract

Imitation learning, and robot learning in general, emerged due to breakthroughs in machine learning, rather than breakthroughs in robotics. As such, evaluation metrics for robot learning are deeply rooted in those for machine learning, and focus primarily on data efficiency. We believe that a better metric for real-world robot learning is time efficiency, which better models the true cost to humans. This is a call to arms to the robot learning community to develop our own evaluation metrics, tailored towards the long-term goals of real-world robotics.