A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models

TL;DR

This study systematically compares different sample-level tokenization strategies for transformer-based neuroimaging models applied to MEG data, finding that simple fixed schemes perform comparably to learnable methods across various evaluation metrics.

Contribution

It introduces a comprehensive evaluation of tokenization methods for neuroimaging data, including a novel autoencoder-based learnable tokenizer, and demonstrates the effectiveness of simple fixed strategies.

Findings

Both learnable and fixed tokenizers achieve high reconstruction fidelity.

Simple fixed tokenization performs comparably to learnable methods on multiple metrics.

Results suggest fixed strategies are sufficient for neural foundation models.

Abstract

Recent success in natural language processing has motivated growing interest in large-scale foundation models for neuroimaging data. Such models often require discretization of continuous neural time series data, a process referred to as 'tokenization'. However, the impact of different tokenization strategies for neural data is currently poorly understood. In this work, we present a systematic evaluation of sample-level tokenization strategies for transformer-based large neuroimaging models (LNMs) applied to magnetoencephalography (MEG) data. We compare learnable and non-learnable tokenizers by examining their signal reconstruction fidelity and their impact on subsequent foundation modeling performance (token prediction, biological plausibility of generated data, preservation of subject-specific information, and performance on downstream tasks). For the learnable tokenizer, we introduce a novel approach based on an autoencoder. Experiments were conducted on three publicly available MEG datasets spanning different acquisition sites, scanners, and experimental paradigms. Our results show that both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most evaluation criteria, suggesting that simple fixed sample-level tokenization strategies can be used in the development of neural foundation models. The code is available at https://github.com/OHBA-analysis/Cho2026_Tokenizer.