User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detection

TL;DR

This paper introduces a novel user-based sequential modeling approach using Transformer encoders for insider threat detection, significantly improving accuracy and reducing false rates by leveraging behavioral sequences.

Contribution

The study proposes a Transformer-based sequential modeling framework for insider threat detection, transforming static datasets into temporal sequences and employing reconstruction errors for anomaly detection.

Findings

Achieved 96.61% accuracy and 99.43% recall on test sets.

Outperformed traditional tabular and autoencoder baselines.

Demonstrated low false negative and false positive rates.

Abstract

Insider threat detection presents unique challenges due to the authorized status of malicious actors and the subtlety of anomalous behaviors. Existing machine learning methods often treat user activity as isolated events, thereby failing to leverage sequential dependencies in user behavior. In this study, we propose a User-Based Sequencing (UBS) methodology, transforming the CERT insider threat dataset into structured temporal sequences suitable for deep sequential modeling. We deploy a Transformer Encoder architecture to model benign user activity and employ its reconstruction errors as anomaly scores. These scores are subsequently evaluated using three unsupervised outlier detection algorithms: One-Class SVM (OCSVM), Local Outlier Factor (LOF), and Isolation Forest (iForest). Across four rigorously designed test sets, including combinations of multiple CERT dataset releases, our UBS-Transformer pipeline consistently achieves state-of-the-art performance - notably 96.61% accuracy, 99.43% recall, 96.38% F1-score, 95.00% AUROC, and exceptionally low false negative (0.0057) and false positive (0.0571) rates. Comparative analyses demonstrate that our approach substantially outperforms tabular and conventional autoencoder baselines, underscoring the efficacy of sequential user modeling and advanced anomaly detection in the insider threat domain.