Toward Privileged Foundation Models:LUPI for Accelerated and Improved Learning

TL;DR

This paper introduces PIQL, a framework that integrates privileged information into tabular foundation models to accelerate training and enhance generalization, reducing data and computational needs.

Contribution

The paper presents a novel method for incorporating privileged information into foundation models, with theoretical analysis and empirical results demonstrating improved efficiency and performance.

Findings

PIQL achieves faster convergence of models.

Models trained with PIQL have lower final loss.

PIQL improves generalization and reduces data requirements.

Abstract

Training foundation models is computationally intensive and often slow to converge. We introduce PIQL,Privileged Information for Quick and Quality Learning, the first framework to systematically integrate privileged information (PI) to simultaneously accelerate learning and improve generalization in tabular foundation models (TFMs). We construct two complementary forms of PI: (i) aggregate dataset-level statistics that reduce the burden on in-context learning, and (ii) encodings of the underlying data-generating program, providing knowledge beyond observable data. We further design an architecture that effectively transfers the train-time-only PI by learning to reconstruct it from observed context at inference. We provide a theoretical analysis characterizing conditions under which PI reduces the population-level approximation gap and accelerates convergence in finite-data regimes. Empirical evidence shows that PIQL enables TFMs to achieve faster convergence, lower final loss, and better generalization, in effect, reducing data and compute requirements. Our work establishes PI-guided pretraining as a principled and practical paradigm for improving the efficiency and performance of foundation models.