Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning

TL;DR

Orion-Bix is a novel tabular foundation model that uses biaxial attention and meta-learning to improve few-shot learning and generalization across diverse tabular datasets, outperforming traditional methods.

Contribution

The paper introduces Orion-Bix, a new model combining biaxial attention and meta-learning for robust, few-shot tabular learning, addressing challenges of mixed data types and limited labeled data.

Findings

Outperforms gradient-boosting baselines on benchmarks.

Remains competitive with state-of-the-art tabular models.

Demonstrates effective transfer of inductive biases across diverse data.

Abstract

Tabular data drive most real-world machine learning applications, yet building general-purpose models for them remains difficult. Mixed numeric and categorical fields, weak feature structure, and limited labeled data make scaling and generalization challenging. To this end, we introduce Orion-Bix, a tabular foundation model that combines biaxial attention with meta-learned in-context reasoning for few-shot tabular learning. Its encoder alternates standard, grouped, hierarchical, and relational attention, fusing their outputs through multi-CLS summarization to capture both local and global dependencies efficiently. A label-aware ICL head adapts on the fly and scales to large label spaces via hierarchical decision routing. Meta-trained on synthetically generated, structurally diverse tables with causal priors, Orion-Bix learns transferable inductive biases across heterogeneous data. Delivered as a scikit-learn compatible foundation model, it outperforms gradient-boosting baselines and remains competitive with state-of-the-art tabular foundation models on public benchmarks, showing that biaxial attention with episodic meta-training enables robust, few-shot-ready tabular learning. The model is publicly available at https://github.com/Lexsi-Labs/Orion-BiX .