A Mutation-based Approach for Assessing Weight Coverage of a Path Planner

TL;DR

This paper introduces a mutation-based method to evaluate if a test suite sufficiently covers all possible decision behaviors of an autonomous car's path planner by mutating its cost function weights.

Contribution

It proposes a novel mutation-based approach to assess weight coverage in the path planner of autonomous vehicles, enhancing scenario-based testing methods.

Findings

Some weights are easier to cover due to common scenario aspects.

More complex scenarios tend to cover more weights.

Preliminary results show the approach's potential in identifying coverage gaps.

Abstract

Autonomous cars are subjected to several different kind of inputs (other cars, road structure, etc.) and, therefore, testing the car under all possible conditions is impossible. To tackle this problem, scenario-based testing for automated driving defines categories of different scenarios that should be covered. Although this kind of coverage is a necessary condition, it still does not guarantee that any possible behaviour of the autonomous car is tested. In this paper, we consider the path planner of an autonomous car that decides, at each timestep, the short-term path to follow in the next few seconds; such decision is done by using a weighted cost function that considers different aspects (safety, comfort, etc.). In order to assess whether all the possible decisions that can be taken by the path planner are covered by a given test suite T, we propose a mutation-based approach that mutates the weights of the cost function and then checks if at least one scenario of T kills the mutant. Preliminary experiments on a manually designed test suite show that some weights are easier to cover as they consider aspects that more likely occur in a scenario, and that more complicated scenarios (that generate more complex paths) are those that allow to cover more weights.