# Advancing codon language modeling with synonymous codon constrained masking

## TL;DR

A new codon language model called SynCodonLM improves DNA sequence modeling by focusing on nucleotide-level patterns through a constraint that limits predictions to synonymous codons.

## Contribution

SynCodonLM introduces a novel constraint that enforces synonymous codon prediction, disentangling codon usage from protein semantics.

## Key findings

- SynCodonLM clusters codons by nucleotide properties rather than amino acid identity.
- The model outperforms existing approaches on six of seven DNA-level benchmarks.
- The approach enables better representation learning for DNA-level biology and synthetic biology applications.

## Abstract

Codon language models offer a promising framework for modeling protein-coding DNA sequences, yet current approaches often conflate codon usage with amino acid semantics, limiting their ability to capture DNA-level biology. We introduce SynCodonLM, a codon language model that enforces a biologically grounded constraint: masked codons are only predicted from synonymous options, guided by the known protein sequence. This design disentangles codon-level from protein-level semantics, enabling the model to learn nucleotide-specific patterns. The constraint is implemented by masking non-synonymous codons from the prediction space prior to softmax. Unlike existing models, which cluster codons by amino acid identity, SynCodonLM clusters by nucleotide properties, revealing structure aligned with DNA-level biology. Furthermore, SynCodonLM outperforms existing models on six of seven benchmarks sensitive to DNA-level features, including messenger RNA and protein expression. Our approach advances domain-specific representation learning and opens avenues for sequence design in synthetic biology, as well as deeper insights into diverse bioprocesses.

Graphical Abstract