Synergy: End-to-end Concept Model

TL;DR

Synergy is an end-to-end concept model that learns to tokenize bytes and effectively bridges different abstraction levels, outperforming traditional tokenizers and showing promising position-independent concept emergence.

Contribution

The paper introduces Synergy, a tokenizer-free, byte-level language model with a learned routing mechanism that enhances abstraction handling and outperforms comparable models like Llama3.

Findings

Synergy produces fewer concept tokens than BBPE tokenizers.

Synergy outperforms Llama3 at the same scale and data.

Position removal improves higher abstraction layer performance.

Abstract

In this paper, we present Synergy, a language model that bridges different levels of abstraction in an end-to-end fashion through a learned routing mechanism. Focusing on low-level linguistic abstraction, we trained our model as a byte-level language model. Our model spontaneously learns to tokenize bytes, producing fewer concept tokens than Byte-level Byte Pair Encoder (BBPE) tokenizers while keeping comparable performance. By comparing with Llama3, we observed an advantage of Synergy under the same model scale and training dataset size. Further studies show that the middle part (the higher abstraction part) of our model performs better when positional encodings are removed, suggesting the emergence of position-independent concepts. These findings demonstrate the feasibility of tokenizer-free architectures, paving the way for more robust and flexible pipelines.