First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models

TL;DR

This paper introduces a first-passage process framework to optimize perturbations in machine learning training, enabling rational design of protocols that enhance training speed and generalization across various models and tasks.

Contribution

It presents a novel first-passage approach to systematically optimize training perturbations, moving beyond ad hoc methods and demonstrating transferability across datasets, architectures, and tasks.

Findings

Identified effective perturbation frequencies for CIFAR-10 with ResNet-18.

Demonstrated transferability of the approach to different datasets and models.

Improved training efficiency and generalization through optimized perturbations.

Abstract

Machine learning models have become indispensable tools in applications across the physical sciences. Their training is often time-consuming, vastly exceeding the inference timescales. Several protocols have been developed to perturb the learning process and improve the training, such as shrink and perturb, warm restarts, and stochastic resetting. For classifiers, these perturbations have been shown to result in enhanced speedups or improved generalization. However, the design of such perturbations is usually done ad hoc by intuition and trial and error. To rationally optimize training protocols, we frame them as first-passage processes and consider their response to perturbations. We show that if the unperturbed learning process reaches a quasi-steady state, the response at a single perturbation frequency can predict the behavior at a wide range of frequencies. We employ this approach to a CIFAR-10 classifier using the ResNet-18 model and identify a useful perturbation and frequency among several possibilities. We demonstrate the transferability of the approach to other datasets, architectures, optimizers and even tasks (regression instead of classification). Our work allows optimization of perturbations for improving the training of machine learning models using a first-passage approach.