CAN-QA: A Question-Answering Benchmark for Reasoning over In-Vehicle CAN Traffic

TL;DR

CAN-QA introduces a novel question-answering benchmark for analyzing in-vehicle CAN traffic, emphasizing reasoning over temporal and relational data rather than simple classification.

Contribution

This work is the first to reformulate CAN traffic analysis as a QA task, creating a large dataset and evaluating language models on complex reasoning challenges.

Findings

Large language models struggle with temporal reasoning in CAN traffic

Models perform well on superficial patterns but poorly on multi-condition inference

CAN-QA dataset enables systematic evaluation of reasoning capabilities in vehicle security

Abstract

The Controller Area Network (CAN) is a safety-critical in-vehicle communication protocol that lacks built-in security mechanisms, making intrusion detection essential. Existing approaches predominantly formulate CAN intrusion detection as a classification task, mapping complex traffic patterns to attack labels. However, this formulation abstracts away the temporal and relational structure of CAN traffic and misaligns with real-world forensic workflows, which require systematic reasoning about traffic behavior. To address this gap, we introduce CAN-QA, the first benchmark that reformulates CAN traffic analysis as a question-answering (QA) task. CAN-QA converts raw CAN logs into temporally segmented windows and applies deterministic rule-based templates to generate natural-language questions paired with automatically derived ground-truth answers. The resulting dataset comprises 33,128 QA pairs across 10 categories, each targeting distinct semantic and temporal properties of CAN traffic. Using CAN-QA, we evaluate large language models across both True/False and multiple-choice formats. Our results indicate that, although these models capture superficial statistical regularities, they struggle with temporal reasoning, multi-condition inference, and higher-level behavioral interpretation. Our code is available at https://github.com/Kriiiiss/CAN-QA.