Learning to Evaluate Autonomous Behaviour in Human-Robot Interaction

TL;DR

This paper introduces a neural meta-evaluator for assessing humanoid robot behaviors in human-robot interaction, providing a systematic, reproducible, and trajectory-based evaluation method for imitation learning policies.

Contribution

The paper presents NeME, a deep learning-based meta-evaluator that assesses robot control policies through trajectory analysis, eliminating the need for human involvement in evaluation.

Findings

NeME aligns well with success rates on the robot.

The framework enables systematic comparison of imitation learning methods.

It improves reproducibility and insightfulness in robot behavior evaluation.

Abstract

Evaluating and comparing the performance of autonomous Humanoid Robots is challenging, as success rate metrics are difficult to reproduce and fail to capture the complexity of robot movement trajectories, critical in Human-Robot Interaction and Collaboration (HRIC). To address these challenges, we propose a general evaluation framework that measures the quality of Imitation Learning (IL) methods by focusing on trajectory performance. We devise the Neural Meta Evaluator (NeME), a deep learning model trained to classify actions from robot joint trajectories. NeME serves as a meta-evaluator to compare the performance of robot control policies, enabling policy evaluation without requiring human involvement in the loop. We validate our framework on ergoCub, a humanoid robot, using teleoperation data and comparing IL methods tailored to the available platform. The experimental results indicate that our method is more aligned with the success rate obtained on the robot than baselines, offering a reproducible, systematic, and insightful means for comparing the performance of multimodal imitation learning approaches in complex HRI tasks.