FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Models

TL;DR

FIRE introduces a training-free multi-fidelity regression framework that leverages tabular foundation models for efficient, accurate, and uncertainty-aware predictions across diverse benchmark problems, outperforming traditional methods.

Contribution

The paper presents FIRE, a novel zero-shot, distribution-conditioned in-context learning approach for multi-fidelity regression using pre-trained tabular foundation models, eliminating the need for retraining.

Findings

Outperforms seven state-of-the-art MF regression methods in accuracy and uncertainty quantification.

Provides a better performance-time trade-off across 31 synthetic and real-world benchmarks.

Achieves robust residual learning without model retraining.

Abstract

Multi-fidelity (MF) regression often operates in regimes of extreme data imbalance, where the commonly-used Gaussian-process (GP) surrogates struggle with cubic scaling costs and overfit to sparse high-fidelity observations, limiting efficiency and generalization in real-world applications. We introduce FIRE, a training-free MF framework that couples tabular foundation models (TFMs) to perform zero-shot in-context Bayesian inference via a high-fidelity correction model conditioned on the low-fidelity model's posterior predictive distributions. This cross-fidelity information transfer via distributional summaries captures heteroscedastic errors, enabling robust residual learning without model retraining. Across 31 benchmark problems spanning synthetic and real-world tasks (e.g., DrivAerNet, LCBench), FIRE delivers a stronger performance-time trade-off than seven state-of-the-art GP-based or deep learning MF regression methods, ranking highest in accuracy and uncertainty quantification with runtime advantages. Limitations include context window constraints and dependence on the quality of the pre-trained TFM's.