Verisimilar Percept Sequences Tests for Autonomous Driving Intelligent Agent Assessment

TL;DR

This paper proposes a systematic method for evaluating perception and decision-making systems in autonomous vehicles, modeling them as intelligent agents and focusing on safety-critical reliability assessment.

Contribution

It introduces a new evaluation framework for autonomous vehicle perception and decision systems, emphasizing safety and reproducibility of AI models.

Findings

Identifies data dependency issues in AI perception models.

Proposes procedures to reproduce AI system behaviors.

Highlights the importance of integrated evaluation methods.

Abstract

The autonomous car technology promises to replace human drivers with safer driving systems. But although autonomous cars can become safer than human drivers this is a long process that is going to be refined over time. Before these vehicles are deployed on urban roads a minimum safety level must be assured. Since the autonomous car technology is still under development there is no standard methodology to evaluate such systems. It is important to completely understand the technology that is being developed to design efficient means to evaluate it. In this paper we assume safety-critical systems reliability as a safety measure. We model an autonomous road vehicle as an intelligent agent and we approach its evaluation from an artificial intelligence perspective. Our focus is the evaluation of perception and decision making systems and also to propose a systematic method to evaluate their integration in the vehicle. We identify critical aspects of the data dependency from the artificial intelligence state of the art models and we also propose procedures to reproduce them.