Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization

TL;DR

This paper introduces QA-Token, a quality-aware tokenization method that improves foundation model pre-training on noisy real-world data by incorporating data reliability into vocabulary construction, leading to significant performance gains.

Contribution

The paper proposes a novel bilevel optimization, reinforcement learning approach, and adaptive parameter learning mechanism for quality-aware tokenization tailored to noisy data.

Findings

6.7% F1 improvement in genomics variant calling

30% Sharpe ratio increase in finance

State-of-the-art pathogen detection at foundation scale

Abstract

Current tokenization methods process sequential data without accounting for signal quality, limiting their effectiveness on noisy real-world corpora. We present QA-Token (Quality-Aware Tokenization), which incorporates data reliability directly into vocabulary construction. We make three key contributions: (i) a bilevel optimization formulation that jointly optimizes vocabulary construction and downstream performance, (ii) a reinforcement learning approach that learns merge policies through quality-aware rewards with convergence guarantees, and (iii) an adaptive parameter learning mechanism via Gumbel-Softmax relaxation for end-to-end optimization. Our experimental evaluation demonstrates consistent improvements: genomics (6.7 percentage point F1 gain in variant calling over BPE), finance (30% Sharpe ratio improvement). At foundation scale, we tokenize a pretraining corpus comprising 1.7 trillion base-pairs and achieve state-of-the-art pathogen detection (94.53 MCC) while reducing token count by 15%. We unlock noisy real-world corpora, spanning petabases of genomic sequences and terabytes of financial time series, for foundation model training with zero inference overhead.