TrimTuner: Efficient Optimization of Machine Learning Jobs in the Cloud via Sub-Sampling

TL;DR

TrimTuner is a system that efficiently optimizes machine learning jobs in the cloud by using sub-sampling techniques, significantly reducing costs and speeding up hyper-parameter recommendations compared to existing methods.

Contribution

It introduces a novel cloud optimization system that leverages sub-sampling and ensemble decision trees to reduce costs and improve speed without full model training.

Findings

Cost reduction of up to 50x in optimization process

Speed-up of 65x in recommendation process

Uses data-sets up to 60x smaller than original

Abstract

This work introduces TrimTuner, the first system for optimizing machine learning jobs in the cloud to exploit sub-sampling techniques to reduce the cost of the optimization process while keeping into account user-specified constraints. TrimTuner jointly optimizes the cloud and application-specific parameters and, unlike state of the art works for cloud optimization, eschews the need to train the model with the full training set every time a new configuration is sampled. Indeed, by leveraging sub-sampling techniques and data-sets that are up to 60x smaller than the original one, we show that TrimTuner can reduce the cost of the optimization process by up to 50x. Further, TrimTuner speeds-up the recommendation process by 65x with respect to state of the art techniques for hyper-parameter optimization that use sub-sampling techniques. The reasons for this improvement are twofold: i) a novel domain specific heuristic that reduces the number of configurations for which the acquisition function has to be evaluated; ii) the adoption of an ensemble of decision trees that enables boosting the speed of the recommendation process by one additional order of magnitude.