Masked-Token Prediction for Anomaly Detection at the Large Hadron Collider

TL;DR

This paper introduces a novel masked-token prediction approach using transformer models for anomaly detection in collider data, effectively identifying subtle deviations indicative of new physics.

Contribution

It applies masked-token prediction with deep-learned tokenization to collider data, demonstrating improved sensitivity and transferability for model-independent anomaly detection.

Findings

Strong performance on four-top-quark signatures.

Deep-learned tokenization outperforms look-up tables.

Model transfers across different BSM searches.

Abstract

Anomaly detection in High Energy Physics requires identifying rare signals against overwhelming backgrounds, without prior knowledge of the signal. We present the first application of masked-token prediction, a technique from Large Language Models, to this problem. A lightweight encoder architecture trained solely on background events captures the structure of Standard Model (SM) physics; at inference, sequences deviating from this learned structure are flagged as anomalous. We evaluate the approach on searches for four-top-quark production and supersymmetric gluino pair production, both featuring top-rich final states with substantial missing transverse energy, covering SM and beyond the Standard Model (BSM) scenarios. Strong performance on the four-top signature, which closely resembles background, demonstrates the method's sensitivity to subtle deviations. We further show that the tokenization strategy significantly impacts performance: deep-learned tokenization via vector-quantized variational autoencoders (VQ-VAE) outperforms look-up table tokenization. Comparison with established anomaly detection baselines confirms robustness. These results highlight the potential of token-based collider data representations combined with transformer architectures for new-physics discovery. Once trained on SM background, the model transfers across different BSM searches, enabling scalable, model-independent anomaly detection at reduced computational cost.