When Cyber-Physical Systems Meet AI: A Benchmark, an Evaluation, and a Way Forward

TL;DR

This paper introduces a public benchmark for AI-enabled cyber-physical systems across seven domains, systematically evaluates AI controllers versus traditional ones, and discusses future research directions for improving performance and reliability.

Contribution

It creates a comprehensive benchmark for industry-level CPS, evaluates AI controllers with state-of-the-art DRL methods, and identifies key challenges and opportunities for hybrid systems and testing techniques.

Findings

AI controllers do not always outperform traditional controllers

Existing CPS testing techniques are insufficient for AI-enabled systems

Hybrid systems combining AI and traditional controllers can improve performance

Abstract

Cyber-physical systems (CPS) have been broadly deployed in safety-critical domains, such as automotive systems, avionics, medical devices, etc. In recent years, Artificial Intelligence (AI) has been increasingly adopted to control CPS. Despite the popularity of AI-enabled CPS, few benchmarks are publicly available. There is also a lack of deep understanding on the performance and reliability of AI-enabled CPS across different industrial domains. To bridge this gap, we initiate to create a public benchmark of industry-level CPS in seven domains and build AI controllers for them via state-of-the-art deep reinforcement learning (DRL) methods. Based on that, we further perform a systematic evaluation of these AI-enabled systems with their traditional counterparts to identify the current challenges and explore future opportunities. Our key findings include (1) AI controllers do not always outperform traditional controllers, (2) existing CPS testing techniques (falsification, specifically) fall short of analyzing AI-enabled CPS, and (3) building a hybrid system that strategically combines and switches between AI controllers and traditional controllers can achieve better performance across different domains. Our results highlight the need for new testing techniques for AI-enabled CPS and the need for more investigations into hybrid CPS systems to achieve optimal performance and reliability.