Fractal Language Modelling by Universal Sequence Maps (USM)

TL;DR

This paper introduces Universal Sequence Maps (USM), a fractal encoding method for symbolic sequences that preserves context at multiple scales, enabling efficient numerical analysis and convergence properties, demonstrated on genomic data.

Contribution

The paper advances bijective fractal encoding with USM by resolving seeding biases and revealing its convergence to a steady state, applicable to sequences of arbitrary alphabet size.

Findings

USM provides a bijective, fractal encoding of sequences.

USM converges to a steady state embedding.

Application demonstrated on genomic sequences with potential for arbitrary alphabets.

Abstract

Motivation: With the advent of Language Models using Transformers, popularized by ChatGPT, there is a renewed interest in exploring encoding procedures that numerically represent symbolic sequences at multiple scales and embedding dimensions. The challenge that encoding addresses is the need for mechanisms that uniquely retain contextual information about the succession of individual symbols, which can then be modeled by nonlinear formulations such as neural networks. Context: Universal Sequence Maps(USM) are iterated functions that bijectively encode symbolic sequences onto embedded numerical spaces. USM is composed of two Chaos Game Representations (CGR), iterated forwardly and backwardly, that can be projected into the frequency domain (FCGR). The corresponding USM coordinates can be used to compute a Chebyshev distance metric as well as k-mer frequencies, without having to recompute the embedded numeric coordinates, and, paradoxically, allowing for non-integers values of k. Results: This report advances the bijective fractal encoding by Universal Sequence Maps (USM) by resolving seeding biases affecting the iterated process. The resolution had two results, the first expected, the second an intriguing outcome: 1) full reconciliation of numeric positioning with sequence identity; and 2) uncovering the nature of USM as an efficient numeric process converging towards a steady state sequence embedding solution. We illustrate these results for genomic sequences because of the convenience of a planar representation defined by an alphabet with only 4 tokens (the 4 nucleotides). Nevertheless, the application to alphabet of arbitrary cardinality was found to be straightforward.