Personalizing black-box models for nonparametric regression with minimax optimality

TL;DR

This paper develops a theoretical framework for adapting pre-trained black-box models to new regression tasks with limited data, achieving minimax optimal rates and demonstrating practical effectiveness.

Contribution

It introduces a novel approach for few-shot personalization of black-box models in nonparametric regression, establishing minimax optimality and robustness guarantees.

Findings

Proposes algorithms that incorporate pre-trained models into regression.

Achieves minimax optimal rates for personalization.

Demonstrates effectiveness through simulations and real data.

Abstract

Recent advances in large-scale models, including deep neural networks and large language models, have substantially improved performance across a wide range of learning tasks. The widespread availability of such pre-trained models creates new opportunities for data-efficient statistical learning, provided they can be effectively integrated into downstream tasks. Motivated by this setting, we study few-shot personalization, where a pre-trained black-box model is adapted to a target domain using a limited number of samples. We develop a theoretical framework for few-shot personalization in nonparametric regression and propose algorithms that can incorporate a black-box pre-trained model into the regression procedure. We establish the minimax optimal rate for the personalization problem and show that the proposed method attains this rate. Our results clarify the statistical benefits of leveraging pre-trained models under sample scarcity and provide robustness guarantees when the pre-trained model is not informative. We illustrate the finite-sample performance of the methods through simulations and an application to the California housing dataset with several pre-trained models.