Control of Overfitting with Physics

TL;DR

This paper explores theoretical explanations for overfitting control in machine learning by drawing analogies from physics and biology, providing insights into algorithm stability and GAN behavior.

Contribution

It introduces physics and biology analogies to explain overfitting mechanisms and control strategies in machine learning models.

Findings

Eyring formula helps control overfitting in Langevin dynamics.

Wide minima with low free energy relate to low overfitting.

Biological predator-prey analogy explains GAN overfitting reduction.

Abstract

While there are many works on the applications of machine learning, not so many of them are trying to understand the theoretical justifications to explain their efficiency. In this work, overfitting control (or generalization property) in machine learning is explained using analogies from physics and biology. For stochastic gradient Langevin dynamics, we show that the Eyring formula of kinetic theory allows to control overfitting in the algorithmic stability approach - when wide minima of the risk function with low free energy correspond to low overfitting. For the generative adversarial network (GAN) model, we establish an analogy between GAN and the predator-prey model in biology. An application of this analogy allows us to explain the selection of wide likelihood maxima and overfitting reduction for GANs.