A Methodology for Developing Foundational Transformer Models in Collider Physics Analysis

TL;DR

This paper introduces a universal transformer-based methodology for analyzing collider physics data, capable of handling multiple processes and final states simultaneously, reducing the need for separate analysis frameworks.

Contribution

It presents a multi-task pre-training approach that captures diverse physics patterns in collider data within a single foundational model, enabling unified analysis across various processes.

Findings

Model maintains sensitivity to rare topologies like 3t and 4t.

Unified architecture reduces analysis complexity.

Pre-training captures cross-process correlations.

Abstract

We present a methodology for training foundational transformer models capable of processing collider data with diverse kinematic signatures. Our universal foundation model is designed for simultaneous analysis of all processes involving from one to four top-quarks production with their corresponding background processes. The approach employs multi-task pre-training on combined datasets of simulated events, enabling the model to capture the full spectrum of interaction physics while extracting universal patterns across different final states prior to task-specific fine-tuning. This unified architecture eliminates the need for separate analysis frameworks for different final signatures and specific tasks. The transformer-based pre-training strategy explicitly preserves unique interaction patterns through adaptive attention mechanisms while establishing cross-process correlations. We plan to demonstrate how this architecture maintains sensitivity to rare high-multiplicity topologies (3t and 4t) without compromising performance on conventional channels ($t\bar{t}$, $tX$, $t\bar{t}H$), effectively bridging the gap between disparate analysis paradigms in collider physics.