Unreasonable Effectivness of Deep Learning

TL;DR

This paper reveals that well-known backpropagation rules can be derived from a weighted combination of finite automata, providing a theoretical framework that links deep learning performance to online statistical bounds and automata theory.

Contribution

It introduces a novel framework connecting finite automata, weighted majority algorithms, and backpropagation, offering new insights into deep learning's effectiveness and network topology design.

Findings

Backpropagation rules arise from automata combinations.

The aggregated automata prediction is equivalent to neural network backpropagation.

Provides bounds on deep learning performance using online statistical results.

Abstract

We show how well known rules of back propagation arise from a weighted combination of finite automata. By redefining a finite automata as a predictor we combine the set of all $k$-state finite automata using a weighted majority algorithm. This aggregated prediction algorithm can be simplified using symmetry, and we prove the equivalence of an algorithm that does this. We demonstrate that this algorithm is equivalent to a form of a back propagation acting in a completely connected $k$-node neural network. Thus the use of the weighted majority algorithm allows a bound on the general performance of deep learning approaches to prediction via known results from online statistics. The presented framework opens more detailed questions about network topology; it is a bridge to the well studied techniques of semigroup theory and applying these techniques to answer what specific network topologies are capable of predicting. This informs both the design of artificial networks and the exploration of neuroscience models.