Towards Meta-learned Algorithm Selection using Implicit Fidelity Information

TL;DR

This paper introduces IMFAS, a meta-learning approach that uses multi-fidelity landmarking and LSTMs to efficiently select algorithms by capturing dataset topology and algorithm biases without full training.

Contribution

IMFAS is a novel method that leverages non-parametric, meta-learned learning curves from candidate algorithms to improve algorithm selection with reduced computational cost.

Findings

IMFAS outperforms Successive Halving with up to 50% of the fidelity sequence.

It produces informative landmarks that enhance ranking accuracy.

The approach efficiently captures dataset topology and algorithm biases.

Abstract

Automatically selecting the best performing algorithm for a given dataset or ranking multiple algorithms by their expected performance supports users in developing new machine learning applications. Most approaches for this problem rely on pre-computed dataset meta-features and landmarking performances to capture the salient topology of the datasets and those topologies that the algorithms attend to. Landmarking usually exploits cheap algorithms not necessarily in the pool of candidate algorithms to get inexpensive approximations of the topology. While somewhat indicative, hand-crafted dataset meta-features and landmarks are likely insufficient descriptors, strongly depending on the alignment of the topologies that the landmarks and the candidate algorithms search for. We propose IMFAS, a method to exploit multi-fidelity landmarking information directly from the candidate algorithms in the form of non-parametrically non-myopic meta-learned learning curves via LSTMs in a few-shot setting during testing. Using this mechanism, IMFAS jointly learns the topology of the datasets and the inductive biases of the candidate algorithms, without the need to expensively train them to convergence. Our approach produces informative landmarks, easily enriched by arbitrary meta-features at a low computational cost, capable of producing the desired ranking using cheaper fidelities. We additionally show that IMFAS is able to beat Successive Halving with at most 50% of the fidelity sequence during test time.