Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning

TL;DR

Orion-MSP introduces a multi-scale sparse attention architecture for tabular in-context learning, effectively capturing hierarchical feature interactions and scaling efficiently to high-dimensional data, outperforming existing models.

Contribution

It proposes a novel multi-scale sparse attention mechanism with block-sparse patterns and a Perceiver-style memory, addressing key limitations of prior tabular ICL models.

Findings

Matches or surpasses state-of-the-art performance on diverse benchmarks.

Scales effectively to high-dimensional tables.

Establishes a new standard for efficient tabular in-context learning.

Abstract

Tabular data remain the predominant format for real-world applications. Yet, developing effective neural models for tabular data remains challenging due to heterogeneous feature types and complex interactions occurring at multiple scales. Recent advances in tabular in-context learning (ICL), such as TabPFN and TabICL, have achieved state-of-the-art performance comparable to gradient-boosted trees (GBTs) without task-specific fine-tuning. However, current architectures exhibit key limitations: (1) single-scale feature processing that overlooks hierarchical dependencies, (2) dense attention with quadratic scaling in table width, and (3) strictly sequential component processing that prevents iterative representation refinement and cross-component communication. To address these challenges, we introduce Orion-MSP, a tabular ICL architecture featuring three key innovations: (1) multi-scale processing to capture hierarchical feature interactions; (2) block-sparse attention combining windowed, global, and random patterns for scalable efficiency and long-range connectivity; and (3) a Perceiver-style memory enabling safe bidirectional information flow across components. Across diverse benchmarks, Orion-MSP matches or surpasses state-of-the-art performance while scaling effectively to high-dimensional tables, establishing a new standard for efficient tabular in-context learning. The model is publicly available at https://github.com/Lexsi-Labs/Orion-MSP .