Hierarchical Collaborative Hyper-parameter Tuning

TL;DR

This paper introduces a hierarchical multi-agent system for distributed hyper-parameter tuning, demonstrating improved performance over traditional methods in machine learning and optimization tasks, especially in high-dimensional spaces.

Contribution

It presents a novel hierarchical agent-based architecture for hyper-parameter tuning, combining distributed search with guided randomization to enhance efficiency and accuracy.

Findings

Outperforms baseline randomized tuning strategies in classification error.

Reduces number of function evaluations needed for optimal hyper-parameters.

Effective in high-dimensional hyper-parameter spaces.

Abstract

Hyper-parameter Tuning is among the most critical stages in building machine learning solutions. This paper demonstrates how multi-agent systems can be utilized to develop a distributed technique for determining near-optimal values for any arbitrary set of hyper-parameters in a machine learning model. The proposed method employs a distributedly formed hierarchical agent-based architecture for the cooperative searching procedure of tuning hyper-parameter values. The presented generic model is used to develop a guided randomized agent-based tuning technique, and its behavior is investigated in both machine learning and global function optimization applications. According the empirical results, the proposed model outperformed both of its underlying randomized tuning strategies in terms of classification error and function evaluations, notably in higher number of dimensions.