Assessing Generalisation Capability of Machine Learning Models for Intrusion Detection

TL;DR

This paper evaluates the generalisation ability of machine learning models for intrusion detection across different datasets, revealing significant performance drops in cross-dataset scenarios and emphasizing the need for adaptive models.

Contribution

It provides an empirical analysis of the generalisation gap in ML-based intrusion detection, highlighting the limitations of current models across diverse network environments.

Findings

Random Forest achieved over 95% accuracy on same dataset

Performance dropped below 40% in cross-dataset testing

Significant generalisation gap identified in intrusion detection models

Abstract

The growth of networked and IoT systems has intensified cyber-security threats and exposed the limits of traditional signature-based intrusion detection. Although machine-learning-based intrusion detection systems often report strong benchmark performance, high ac- curacy within a single dataset does not necessarily guarantee reliable performance in unseen network environments. This study investigates the generalisation capability of supervised machine learning models for intrusion detection using UNSW-NB15 and TON_IoT. Random Forest, Logistic Regression, and Naive Bayes were evaluated under same-dataset and cross-dataset settings. Random Forest achieved the strongest same dataset performance, with 95.08% accuracy on UNSW-NB15 and 99.79% on TON_IoT, but performance dropped sharply in cross-dataset testing. When trained on UNSW-NB15 and tested on TON_IoT or vice versa, below 40% accuracy. These results reveal a significant generalisation gap in intrusion detection. We connect this challenge to affective computing and human-centric AI, where behavioural signal analysis, anomaly detection, domain shift, and context-sensitive modelling are also central. This framing highlights the need for adaptive, generalisable cyber-security models that can operate across changing network and IoT environments.