DNAMotifTokenizer: Towards Biologically Informed Tokenization of Genomic Sequences

TL;DR

This paper introduces DNAMotifTokenizer, a biologically informed tokenization method for DNA sequences that improves genomic language models by incorporating domain knowledge of DNA motifs, leading to better performance and interpretability.

Contribution

We propose DNAMotifTokenizer, a novel tokenization approach that integrates DNA motif knowledge, outperforming traditional methods like BPE in genomic tasks.

Findings

BPE performs well on small biologically relevant data

Tokenizer choice affects task-specific performance

Knowledge-infused tokenization enhances model interpretability

Abstract

DNA language models have advanced genomics, but their downstream performance varies widely due to differences in tokenization, pretraining data, and architecture. We argue that a major bottleneck lies in tokenizing sparse and unevenly distributed DNA sequence motifs, which are critical for accurate and interpretable models. To investigate, we systematically benchmark k-mer and Byte-Pair Encoding (BPE) tokenizers under controlled pretraining budget, evaluating across multiple downstream tasks from five datasets. We find that tokenizer choice induces task-specific trade-offs, and that vocabulary size and tokenizer training data strongly influence the biological knowledge captured. Notably, BPE tokenizers achieve strong performance when trained on smaller but biologically significant data. Building on these insights, we introduce DNAMotifTokenizer, which directly incorporates domain knowledge of DNA sequence motifs into the tokenization process. DNAMotifTokenizer consistently outperforms BPE across diverse benchmarks, demonstrating that knowledge-infused tokenization is crucial for learning powerful, interpretable, and generalizable genomic representations.