Refined Kolmogorov Complexity of Analog, Evolving and Stochastic Recurrent Neural Networks

TL;DR

This paper characterizes the computational power of analog, evolving, and stochastic neural networks using Kolmogorov complexity, establishing hierarchies between classical complexity classes and providing a unified framework for their capabilities.

Contribution

It introduces a refined hierarchy of complexity classes for different neural network types based on Kolmogorov complexity, bridging gaps between known computational classes.

Findings

Hierarchies of analog, evolving, and stochastic networks are established.

These hierarchies lie between P and P/poly, and BPP and BPP/log*.

A generic construction method for these hierarchies is described.

Abstract

We provide a refined characterization of the super-Turing computational power of analog, evolving, and stochastic neural networks based on the Kolmogorov complexity of their real weights, evolving weights, and real probabilities, respectively. First, we retrieve an infinite hierarchy of classes of analog networks defined in terms of the Kolmogorov complexity of their underlying real weights. This hierarchy is located between the complexity classes $\mathbf{P}$ and $\mathbf{P/poly}$. Then, we generalize this result to the case of evolving networks. A similar hierarchy of Kolomogorov-based complexity classes of evolving networks is obtained. This hierarchy also lies between $\mathbf{P}$ and $\mathbf{P/poly}$. Finally, we extend these results to the case of stochastic networks employing real probabilities as source of randomness. An infinite hierarchy of stochastic networks based on the Kolmogorov complexity of their probabilities is therefore achieved. In this case, the hierarchy bridges the gap between $\mathbf{BPP}$ and $\mathbf{BPP/log^*}$. Beyond proving the existence and providing examples of such hierarchies, we describe a generic way of constructing them based on classes of functions of increasing complexity. For the sake of clarity, this study is formulated within the framework of echo state networks. Overall, this paper intends to fill the missing results and provide a unified view about the refined capabilities of analog, evolving and stochastic neural networks.