CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code

TL;DR

This paper explores how different subtokenization strategies affect large language model pretraining on source code, aiming to optimize length efficiency and model performance.

Contribution

It introduces a subtokenization method that reduces sequence length by 17% without performance loss and demonstrates potential quality improvements with careful subtokenization choices.

Findings

Subtokenization reduces sequence length by 17%.

Careful subtokenization can improve model quality by 0.5-2%.

Length efficiency and performance can be balanced through subtokenization.

Abstract

Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, namely the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account code specifics. We propose subtokenziation that reduces average length by 17% without downstream performance drop, and show that a carefully chosen subtokenization may improve quality by 0.5-2%, possibly with some length increase.