Investigating 1-Bit Quantization in Transformer-Based Top Tagging

TL;DR

This paper introduces BitParT, a 1-bit Transformer model for top-quark tagging in high-energy physics, enabling efficient, low-resource deployment while maintaining competitive accuracy.

Contribution

The paper presents a novel binary-weight Transformer architecture tailored for HEP top-quark tagging, extending ultra-low-bit LLM ideas to physics applications.

Findings

BitParT achieves competitive performance with full-precision models.

Significant reduction in model size and computational complexity.

Demonstrates feasibility of real-time inference in collider experiments.

Abstract

The increasing scale of deep learning models in high-energy physics (HEP) has posed challenges to their deployment on low-power, latency-sensitive platforms, such as FPGAs and ASICs used in trigger systems, as well as in offline data reconstruction and processing pipelines. In this work, we introduce BitParT, a 1-bit Transformer-based architecture designed specifically for the top-quark tagging method. Building upon recent advances in ultra-low-bit large language models (LLMs), we extended these ideas to the HEP domain by developing a binary-weight variant (BitParT) of the Particle Transformer (ParT) model. Our findings indicate a potential for substantial reduction in model size and computational complexity, while maintaining high tagging performance. We benchmark BitParT on the public Top Quark Tagging Reference Dataset and show that it achieves competitive performance relative to its full-precision counterpart. This work demonstrates the design of extreme quantized models for physics applications, paving the way for real-time inference in collider experiments with minimal and optimized resource usage.