Evaluating False Alarm and Missing Attacks in CAN IDS

TL;DR

This paper systematically evaluates the robustness of machine learning-based CAN intrusion detection systems against adversarial attacks, revealing vulnerabilities that can cause missed detections and false alarms in automotive security.

Contribution

It provides a comprehensive adversarial evaluation of various ML-based CAN IDS using the ROAD dataset and protocol-compliant perturbations, highlighting their vulnerabilities and robustness differences.

Findings

All models are vulnerable to adversarial attacks causing missed detections.

Deep neural networks perform best on benign traffic but are still vulnerable.

Shallow models like extra trees show improved robustness against certain attacks.

Abstract

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network-based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missed-attack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.