Performance comparison of timing-based anomaly detectors for Controller Area Network: a reproducible study

TL;DR

This paper provides a comprehensive, reproducible comparison of eight timing-based anomaly detection algorithms for CAN bus in vehicles, addressing previous limitations of non-public datasets and lack of open implementations.

Contribution

It offers the first open, reproducible evaluation of multiple CAN anomaly detectors, including datasets and code, enabling fair comparison and scientific progress.

Findings

Timing-based detectors vary in detection performance

Open datasets and implementations facilitate reproducibility

Benchmark results highlight strengths and weaknesses of each algorithm

Abstract

This work presents an experimental evaluation of the detection performance of eight different algorithms for anomaly detection on the Controller Area Network (CAN) bus of modern vehicles based on the analysis of the timing or frequency of CAN messages. This work solves the current limitations of related scientific literature, that is based on private dataset, lacks of open implementations, and detailed description of the detection algorithms. These drawback prevent the reproducibility of published results, and makes it impossible to compare a novel proposal against related work, thus hindering the advancement of science. This paper solves these issues by publicly releasing implementations, labeled datasets and by describing an unbiased experimental comparisons.