OpenCat: Improving Interoperability of ADS Testing

TL;DR

This paper introduces OpenCat, a converter that enhances the interoperability of ADAS testing benchmarks by transforming test cases into widely supported formats, enabling unbiased and reusable evaluation of ADAS models across different simulators.

Contribution

OpenCat provides a novel conversion tool that improves the reusability and transferability of ADAS benchmarks across various simulation platforms.

Findings

High accuracy in converting OpenDRIVE to Catmull-Rom spline format.

Converted scenarios successfully evaluated a lane-keeping ADAS model.

Benchmarks tailored to specific ADAS models limit their broader usability.

Abstract

Testing Advanced Driving Assistance Systems (ADAS), such as lane-keeping functions, requires creating road topologies or using predefined benchmarks. However, the test cases in existing ADAS benchmarks are often designed in specific formats (e.g., OpenDRIVE) and tailored to specific ADAS models. This limits their reusability and interoperability with other simulators and models, making it challenging to assess ADAS functionalities independently of the platform-specific details used to create the test cases. This paper evaluates the interoperability of SensoDat, a benchmark developed for ADAS regression testing. We introduce OpenCat, a converter that transforms OpenDRIVE test cases into the Catmull-Rom spline format, which is widely supported by many current test generators. By applying OpenCat to the SensoDat dataset, we achieved high accuracy in converting test cases into reusable road scenarios. To validate the converted scenarios, we used them to evaluate a lane-keeping ADAS model using the Udacity simulator. Both the simulator and the ADAS model operate independently of the technologies underlying SensoDat, ensuring an unbiased evaluation of the original test cases. Our findings reveal that benchmarks built with specific ADAS models hinder their effective usage for regression testing. We conclude by offering insights and recommendations to enhance the reusability and transferability of ADAS benchmarks for more extensive applications.