Trustworthy Evaluation of Robotic Manipulation: A New Benchmark and AutoEval Methods

TL;DR

This paper introduces a new benchmark and auto-evaluation methods for trustworthy robotic manipulation assessment, addressing trust dimensions like source authenticity and execution quality, with high accuracy in policy versus teleoperation discrimination.

Contribution

It presents Eval-Actions benchmark and AutoEval architecture, integrating diverse data and signals for comprehensive, trustworthy evaluation of robotic policies.

Findings

AutoEval achieves SRCC of 0.81 and 0.84 under different protocols.

The framework can distinguish policy-generated from teleoperated videos with 99.6% accuracy.

The dataset includes failure scenarios and multiple supervision signals for robust evaluation.

Abstract

Driven by the rapid evolution of Vision-Action and Vision-Language-Action models, imitation learning has significantly advanced robotic manipulation capabilities. However, evaluation methodologies have lagged behind, hindering the establishment of Trustworthy Evaluation for these behaviors. Current paradigms rely on binary success rates, failing to address the critical dimensions of trust: Source Authenticity (i.e., distinguishing genuine policy behaviors from human teleoperation) and Execution Quality (e.g., smoothness and safety). To bridge these gaps, we propose a solution that combines the Eval-Actions benchmark and the AutoEval architecture. First, we construct the Eval-Actions benchmark to support trustworthiness analysis. Distinct from existing datasets restricted to successful human demonstrations, Eval-Actions integrates VA and VLA policy execution trajectories alongside human teleoperation data, explicitly including failure scenarios. This dataset is structured around three core supervision signals: Expert Grading (EG), Rank-Guided preferences (RG), and Chain-of-Thought (CoT). Building on this, we propose the AutoEval architecture: AutoEval leverages Spatio-Temporal Aggregation for semantic assessment, augmented by an auxiliary Kinematic Calibration Signal to refine motion smoothness; AutoEval Plus (AutoEval-P) incorporates the Group Relative Policy Optimization (GRPO) paradigm to enhance logical reasoning capabilities. Experiments show AutoEval achieves Spearman's Rank Correlation Coefficients (SRCC) of 0.81 and 0.84 under the EG and RG protocols, respectively. Crucially, the framework possesses robust source discrimination capabilities, distinguishing between policy-generated and teleoperated videos with 99.6% accuracy, thereby establishing a rigorous standard for trustworthy robotic evaluation. Our project and code are available at https://term-bench.github.io/.