Innovative tokenisation of structured data for LLM training

TL;DR

This paper presents a hybrid tokenisation method for converting structured tabular data into a sequential format suitable for Large Language Model training, effectively handling mixed data types and preserving table structure.

Contribution

It introduces a novel tokenisation approach combining fixed tokens and learned subword vocabularies, enabling efficient processing of large-scale structured datasets for LLMs.

Findings

Processed over 31 million network flows in under five hours

Achieved a data compression ratio of 6.18:1

Created a corpus of over one billion tokens for LLM training

Abstract

Data representation remains a fundamental challenge in machine learning, particularly when adapting sequence-based architectures like Transformers and Large Language Models (LLMs) for structured tabular data. Existing methods often fail to cohesively encode the mix of numerical and categorical features or preserve the inherent structure of tables. This paper introduces a novel, hybrid tokenisation methodology designed to convert tabular data into a unified, sequential format suitable for LLM training. Our approach combines predefined fixed tokens to represent structural elements and low-cardinality categorical features, with a learned subword vocabulary using Byte-Pair Encoding (BPE) for high-cardinality and continuous values. We demonstrate the efficacy of this technique by applying it to a large-scale NetFlow dataset (CIDDS-001), preparing a corpus for a Network Intrusion Detection System (NIDS) foundation model. The evaluation shows that our method is highly efficient, processing over 31 million network flows in under five hours and achieving a significant data compression ratio of 6.18:1. This process resulted in a computationally manageable corpus of over one billion tokens, establishing a viable and generalisable pathway for training foundation models on structured data.