A Protocol for Validating Social Navigation Policies

TL;DR

This paper introduces a standardized protocol for benchmarking social navigation policies in robots, using canonical scenarios and questionnaires to evaluate social behavior in a realistic, scalable, and repeatable manner.

Contribution

It proposes a novel benchmarking protocol for social navigation that enables consistent evaluation across different scenarios and setups.

Findings

Protocol is realistic and scalable

Benchmarking is repeatable across runs and spaces

Can be used for new social navigation scenarios

Abstract

Enabling socially acceptable behavior for situated agents is a major goal of recent robotics research. Robots should not only operate safely around humans, but also abide by complex social norms. A key challenge for developing socially-compliant policies is measuring the quality of their behavior. Social behavior is enormously complex, making it difficult to create reliable metrics to gauge the performance of algorithms. In this paper, we propose a protocol for social navigation benchmarking that defines a set of canonical social navigation scenarios and an in-situ metric for evaluating performance on these scenarios using questionnaires. Our experiments show this protocol is realistic, scalable, and repeatable across runs and physical spaces. Our protocol can be replicated verbatim or it can be used to define a social navigation benchmark for novel scenarios. Our goal is to introduce a protocol for benchmarking social scenarios that is homogeneous and comparable.