Scaling Particle Collision Data Analysis

TL;DR

This paper introduces BBT-Neutron, a novel large language model with binary tokenization designed for large-scale numerical data analysis in particle physics, demonstrating competitive performance in jet origin classification.

Contribution

The paper presents BBT-Neutron, a task-agnostic LLM with binary tokenization that effectively handles numerical scientific data, extending LLM capabilities to scientific big data analysis.

Findings

BBT-Neutron achieves performance comparable to specialized models in jet origin identification.

Binary tokenization improves LLM handling of large-scale numerical data.

Scaling data volume enhances BBT-Neutron's performance, indicating its potential as a scientific foundation model.

Abstract

For decades, researchers have developed task-specific models to address scientific challenges across diverse disciplines. Recently, large language models (LLMs) have shown enormous capabilities in handling general tasks; however, these models encounter difficulties in addressing real-world scientific problems, particularly in domains involving large-scale numerical data analysis, such as experimental high energy physics. This limitation is primarily due to BPE tokenization's inefficacy with numerical data. In this paper, we propose a task-agnostic architecture, BBT-Neutron, which employs a binary tokenization method to facilitate pretraining on a mixture of textual and large-scale numerical experimental data. We demonstrate the application of BBT-Neutron to Jet Origin Identification (JoI), a critical categorization challenge in high-energy physics that distinguishes jets originating from various quarks or gluons. Our results indicate that BBT-Neutron achieves comparable performance to state-of-the-art task-specific JoI models. Furthermore, we examine the scaling behavior of BBT-Neutron's performance with increasing data volume, suggesting the potential for BBT-Neutron to serve as a foundational model for particle physics data analysis, with possible extensions to a broad spectrum of scientific computing applications for Big Science experiments, industrial manufacturing and spacial computing. The project code is available at https://github.com/supersymmetry-technologies/bbt-neutron.