Negative Feedback System as Optimizer for Machine Learning Systems

TL;DR

This paper explores how negative feedback systems can serve as physical analogs for optimization in machine learning, capable of learning non-differentiable functions and offering new perspectives on neural network training.

Contribution

It introduces a novel analogy between negative feedback systems and machine learning optimization, deriving backpropagation and other techniques independently of gradient descent.

Findings

Negative feedback can learn non-differentiable functions.

Optimization reduces to gradient descent under squared error.

Provides a new physical perspective on neural network training.

Abstract

With high forward gain, a negative feedback system has the ability to perform the inverse of a linear or non-linear function that is in the feedback path. This property of negative feedback systems has been widely used in analog electronic circuits to construct precise closed-loop functions. This paper describes how the function-inverting process of a negative feedback system serves as a physical analogy of the optimization technique in machine learning. We show that this process is able to learn some non-differentiable functions in cases where a gradient descent-based method fails. We also show that the optimization process reduces to gradient descent under the constraint of squared error minimization. We derive the backpropagation technique and other known optimization techniques of deep networks from the properties of negative feedback system independently of the gradient descent method. This analysis provides a novel view of neural network optimization and may provide new insights on open problems.