Towards Understanding Layer Contributions in Tabular In-Context Learning Models

TL;DR

This paper explores how individual layers in tabular in-context learning models contribute to predictions, revealing redundancies and similarities with large language models, which can inform model compression and interpretability.

Contribution

It provides the first detailed analysis of layer-wise contributions in tabular ICL models, highlighting redundancies and comparing dynamics with LLMs.

Findings

Only subsets of layers share common representations

Identification of redundant layers suggests potential for model compression

Layer dynamics in tabular ICL models are similar to those in LLMs

Abstract

Despite the architectural similarities between tabular in-context learning (ICL) models and large language models (LLMs), little is known about how individual layers contribute to tabular prediction. In this paper, we investigate how the latent spaces evolve across layers in tabular ICL models, identify potential redundant layers, and compare these dynamics with those observed in LLMs. We analyze TabPFN and TabICL through the "layers as painters" perspective, finding that only subsets of layers share a common representational language, suggesting structural redundancy and offering opportunities for model compression and improved interpretability.