TRUST-TECH based Methods for Optimization and Learning

TL;DR

This paper introduces TRUST-TECH based methods that systematically explore parameter spaces to find multiple local optima, improving optimization and learning in machine learning by reducing initialization sensitivity and integrating with other algorithms.

Contribution

It proposes a novel TRUST-TECH framework for optimization in machine learning, combining local and neighborhood search stages, and integrating with other stochastic algorithms.

Findings

Reduces sensitivity to initialization in optimization tasks.

Enhances solution quality by systematic exploration of local optima.

Demonstrates effectiveness on synthetic and real datasets.

Abstract

Many problems that arise in machine learning domain deal with nonlinearity and quite often demand users to obtain global optimal solutions rather than local optimal ones. Optimization problems are inherent in machine learning algorithms and hence many methods in machine learning were inherited from the optimization literature. Popularly known as the initialization problem, the ideal set of parameters required will significantly depend on the given initialization values. The recently developed TRUST-TECH (TRansformation Under STability-reTaining Equilibria CHaracterization) methodology systematically explores the subspace of the parameters to obtain a complete set of local optimal solutions. In this thesis work, we propose TRUST-TECH based methods for solving several optimization and machine learning problems. Two stages namely, the local stage and the neighborhood-search stage, are repeated alternatively in the solution space to achieve improvements in the quality of the solutions. Our methods were tested on both synthetic and real datasets and the advantages of using this novel framework are clearly manifested. This framework not only reduces the sensitivity to initialization, but also allows the flexibility for the practitioners to use various global and local methods that work well for a particular problem of interest. Other hierarchical stochastic algorithms like evolutionary algorithms and smoothing algorithms are also studied and frameworks for combining these methods with TRUST-TECH have been proposed and evaluated on several test systems.