Hybrid Tokenization Strategy for DNA Language Model using Byte Pair Encoding and K-MER Methods

TL;DR

This paper introduces a hybrid tokenization approach combining 6-mer and BPE methods to improve DNA language models, capturing both local and global sequence features for better prediction accuracy.

Contribution

The study proposes a novel hybrid tokenization strategy that merges k-mer and BPE tokens, enhancing DNA language model performance over existing methods.

Findings

Achieved higher next-k-mer prediction accuracy than state-of-the-art models.

Demonstrated improved capture of local and global DNA sequence features.

Validated the effectiveness of hybrid tokenization in genomic language modeling.

Abstract

This paper presents a novel hybrid tokenization strategy that enhances the performance of DNA Language Models (DLMs) by combining 6-mer tokenization with Byte Pair Encoding (BPE-600). Traditional k-mer tokenization is effective at capturing local DNA sequence structures but often faces challenges, including uneven token distribution and a limited understanding of global sequence context. To address these limitations, we propose merging unique 6mer tokens with optimally selected BPE tokens generated through 600 BPE cycles. This hybrid approach ensures a balanced and context-aware vocabulary, enabling the model to capture both short and long patterns within DNA sequences simultaneously. A foundational DLM trained on this hybrid vocabulary was evaluated using next-k-mer prediction as a fine-tuning task, demonstrating significantly improved performance. The model achieved prediction accuracies of 10.78% for 3-mers, 10.1% for 4-mers, and 4.12% for 5-mers, outperforming state-of-the-art models such as NT, DNABERT2, and GROVER. These results highlight the ability of the hybrid tokenization strategy to preserve both the local sequence structure and global contextual information in DNA modeling. This work underscores the importance of advanced tokenization methods in genomic language modeling and lays a robust foundation for future applications in downstream DNA sequence analysis and biological research.