Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

TL;DR

This paper introduces a scalable Gaussian process model with latent Kronecker structure for efficient learning curve prediction in AutoML, handling missing data and reducing computational complexity significantly.

Contribution

It proposes a novel latent Kronecker structure for Gaussian processes that enables efficient inference on large-scale learning curve data with missing values.

Findings

Achieves $ ext{O}(n^3 + m^3)$ time complexity, significantly faster than naive methods.

Matches Transformer performance on learning curve prediction tasks.

Effectively handles missing learning curve observations with structured covariance.

Abstract

A key task in AutoML is to model learning curves of machine learning models jointly as a function of model hyper-parameters and training progression. While Gaussian processes (GPs) are suitable for this task, na\"ive GPs require $\mathcal{O}(n^3m^3)$ time and $\mathcal{O}(n^2 m^2)$ space for $n$ hyper-parameter configurations and $\mathcal{O}(m)$ learning curve observations per hyper-parameter. Efficient inference via Kronecker structure is typically incompatible with early-stopping due to missing learning curve values. We impose $\textit{latent Kronecker structure}$ to leverage efficient product kernels while handling missing values. In particular, we interpret the joint covariance matrix of observed values as the projection of a latent Kronecker product. Combined with iterative linear solvers and structured matrix-vector multiplication, our method only requires $\mathcal{O}(n^3 + m^3)$ time and $\mathcal{O}(n^2 + m^2)$ space. We show that our GP model can match the performance of a Transformer on a learning curve prediction task.