Descriptive complexity for neural networks via Boolean networks

TL;DR

This paper explores the expressive power of neural networks through the lens of descriptive complexity, establishing polynomial relationships and translation methods between neural networks, Boolean networks, and logical frameworks.

Contribution

It introduces a formal translation framework between neural networks and Boolean rule-based logic, revealing polynomial size relations and enabling activation function flexibility.

Findings

Neural networks and Boolean rules are polynomially related in size.

Translations between neural networks and Boolean logic have linear or polylogarithmic time delay.

The framework allows converting neural networks to use any activation function, including linear, via floating-point representations.

Abstract

We investigate the expressive power of neural networks from the point of view of descriptive complexity. We study neural networks that use floating-point numbers and piecewise polynomial activation functions from two perspectives: 1) the general scenario where neural networks run for an unlimited number of rounds and have unrestricted topologies, and 2) classical feedforward neural networks that have the topology of layered acyclic graphs and run for only a constant number of rounds. We characterize these neural networks via Boolean networks formalized via a recursive rule-based logic. In particular, we show that the sizes of the neural networks and the corresponding Boolean rule formulae are polynomially related. In fact, in the direction from Boolean rules to neural networks, the blow-up is only linear. Our translations result in a time delay, which is the number of rounds that it takes to simulate a single computation step. In the translation from neural networks to Boolean rules, the time delay of the resulting formula is polylogarithmic in the size of the neural network. In the converse translation, the time delay of the neural network is linear in the formula size. Ultimately, we obtain translations between neural networks, Boolean networks, the diamond-free fragment of modal substitution calculus, and a class of recursive Boolean circuits. Our translations offer a method, for almost any activation function F, of translating any neural network in our setting into an equivalent neural network that uses F at each node. This even includes linear activation functions, which is possible due to using floats rather than actual reals!