Green Machine Learning via Augmented Gaussian Processes and Multi-Information Source Optimization

TL;DR

This paper introduces AGP-MISO, an augmented Gaussian process method that efficiently utilizes multiple information sources with varying costs and fidelities to optimize machine learning models, reducing computational resources and energy consumption.

Contribution

The paper proposes a novel Augmented Gaussian Process method for multi-source optimization, focusing on reliable information sources and a new acquisition function to improve hyperparameter tuning efficiency.

Findings

AGP-MISO outperforms traditional Bayesian Optimization in hyperparameter tuning.

Using multiple sources reduces computational costs significantly.

The method effectively leverages smaller, cheaper data sources for large dataset optimization.

Abstract

Searching for accurate Machine and Deep Learning models is a computationally expensive and awfully energivorous process. A strategy which has been gaining recently importance to drastically reduce computational time and energy consumed is to exploit the availability of different information sources, with different computational costs and different "fidelity", typically smaller portions of a large dataset. The multi-source optimization strategy fits into the scheme of Gaussian Process based Bayesian Optimization. An Augmented Gaussian Process method exploiting multiple information sources (namely, AGP-MISO) is proposed. The Augmented Gaussian Process is trained using only "reliable" information among available sources. A novel acquisition function is defined according to the Augmented Gaussian Process. Computational results are reported related to the optimization of the hyperparameters of a Support Vector Machine (SVM) classifier using two sources: a large dataset - the most expensive one - and a smaller portion of it. A comparison with a traditional Bayesian Optimization approach to optimize the hyperparameters of the SVM classifier on the large dataset only is reported.