A Framework for Model Search Across Multiple Machine Learning Implementations

TL;DR

This paper introduces a unified framework that simplifies and accelerates the process of exploring multiple machine learning implementations during model search, reducing development effort and improving efficiency.

Contribution

It presents a flexible, scalable framework that allows easy addition of new ML algorithms and efficient handling of large search spaces with minimal coding effort.

Findings

Framework requires only 55-144 lines of code to add new ML implementations.

It is the fastest framework for the HIGGS dataset.

It is the second-fastest framework for the SECOM dataset.

Abstract

Several recently devised machine learning (ML) algorithms have shown improved accuracy for various predictive problems. Model searches, which explore to find an optimal ML algorithm and hyperparameter values for the target problem, play a critical role in such improvements. During a model search, data scientists typically use multiple ML implementations to construct several predictive models; however, it takes significant time and effort to employ multiple ML implementations due to the need to learn how to use them, prepare input data in several different formats, and compare their outputs. Our proposed framework addresses these issues by providing simple and unified coding method. It has been designed with the following two attractive features: i) new machine learning implementations can be added easily via common interfaces between the framework and ML implementations and ii) it can be scaled to handle large model configuration search spaces via profile-based scheduling. The results of our evaluation indicate that, with our framework, implementers need only write 55-144 lines of code to add a new ML implementation. They also show that ours was the fastest framework for the HIGGS dataset, and the second-fastest for the SECOM dataset.