LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection

TL;DR

This paper explores how large language models can serve as in-context meta-learners to recommend models and hyperparameters based on dataset metadata, reducing the need for search and expert input.

Contribution

It introduces a novel approach where LLMs are used for model and hyperparameter selection via in-context learning, demonstrating effectiveness without traditional search methods.

Findings

LLMs can recommend competitive models and hyperparameters using dataset metadata.

Meta-informed prompting improves LLM performance in model selection.

LLMs show capacity for in-context meta-learning across benchmarks.

Abstract

Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization.