Interpretable Tabular Foundation Models via In-Context Kernel Regression

TL;DR

KernelICL enhances tabular foundation models by making their predictions transparent through kernel regression, unifying various methods and maintaining competitive performance across numerous datasets.

Contribution

The paper introduces KernelICL, a novel framework that makes in-context learning in tabular models interpretable by explicitly using kernel functions in the prediction layer.

Findings

KernelICL achieves performance comparable to existing models on 55 datasets.

It provides quantifiable interpretability via weight distribution perplexity.

The framework unifies kernel methods, neighbor approaches, and attention mechanisms.

Abstract

Tabular foundation models like TabPFN and TabICL achieve state-of-the-art performance through in-context learning, yet their architectures remain fundamentally opaque. We introduce KernelICL, a framework to enhance tabular foundation models with quantifiable sample-based interpretability. Building on the insight that in-context learning is akin to kernel regression, we make this mechanism explicit by replacing the final prediction layer with kernel functions (Gaussian, dot-product, kNN) so that every prediction is a transparent weighted average of training labels. We introduce a two-dimensional taxonomy that formally unifies standard kernel methods, modern neighbor-based approaches, and attention mechanisms under a single framework, and quantify inspectability via the perplexity of the weight distribution over training samples. On 55 TALENT benchmark datasets, KernelICL achieves performance on par with existing tabular foundation models, demonstrating that explicit kernel constraints on the final layer enable inspectable predictions without sacrificing performance.