Evaluating Dynamic Environment Difficulty for Obstacle Avoidance Benchmarking

TL;DR

This paper introduces four new metrics to quantify the difficulty of dynamic obstacle environments for autonomous systems, validated through extensive simulation to improve benchmarking and development of obstacle avoidance methods.

Contribution

It proposes a comprehensive set of difficulty metrics for dynamic environments and validates their effectiveness in benchmarking obstacle avoidance performance.

Findings

Survivability metric correlates strongly with success rate (SRCC > 0.9).

Lower survivability indicates higher success rate for planners.

Metrics enable fair benchmarking and insights for method improvement.

Abstract

Dynamic obstacle avoidance is a popular research topic for autonomous systems, such as micro aerial vehicles and service robots. Accurately evaluating the performance of dynamic obstacle avoidance methods necessitates the establishment of a metric to quantify the environment's difficulty, a crucial aspect that remains unexplored. In this paper, we propose four metrics to measure the difficulty of dynamic environments. These metrics aim to comprehensively capture the influence of obstacles' number, size, velocity, and other factors on the difficulty. We compare the proposed metrics with existing static environment difficulty metrics and validate them through over 1.5 million trials in a customized simulator. This simulator excludes the effects of perception and control errors and supports different motion and gaze planners for obstacle avoidance. The results indicate that the survivability metric outperforms and establishes a monotonic relationship between the success rate, with a Spearman's Rank Correlation Coefficient (SRCC) of over 0.9. Specifically, for every planner, lower survivability leads to a higher success rate. This metric not only facilitates fair and comprehensive benchmarking but also provides insights for refining collision avoidance methods, thereby furthering the evolution of autonomous systems in dynamic environments.