Contrast Sets for Evaluating Language-Guided Robot Policies

TL;DR

This paper introduces contrast sets as a method to efficiently evaluate language-guided robot policies by applying specific perturbations to test instances, providing more insightful performance analysis with less effort.

Contribution

The work presents contrast sets for robotics as a novel evaluation approach that reduces experimenter effort while offering deeper insights into policy robustness and performance.

Findings

Contrast sets reveal policy weaknesses more effectively.

Evaluation with contrast sets requires less effort than traditional methods.

Applicable to both simulated and real robot tasks.

Abstract

Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances. We use the relative performance change of different contrast set perturbations to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task. We encourage the use of contrast set evaluations as a more informative alternative to small scale, i.i.d. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.