MinMax Recurrent Neural Cascades

TL;DR

MinMax Recurrent Neural Cascades (RNCs) are a novel recurrence model that is highly expressive, computationally efficient, and resistant to vanishing/exploding gradients, with promising empirical results.

Contribution

The paper introduces MinMax RNCs, a new recurrence framework with strong theoretical properties and demonstrated practical capabilities on synthetic and real-world tasks.

Findings

MinMax RNCs can express all regular languages.

They can be evaluated in parallel with logarithmic runtime.

Empirical results show superior performance on synthetic tasks and competitive results on next-token prediction.

Abstract

We show that the MinMax algebra provides a form of recurrence that is expressively powerful, efficiently implementable, and most importantly it is not affected by vanishing or exploding gradient. We call MinMax Recurrent Neural Cascades (RNCs) the models obtained by cascading several layers of neurons that employ such recurrence. We show that MinMax RNCs enjoy many favourable theoretical properties. First, their formal expressivity includes all regular languages, arguably the maximal expressivity for a finite-memory system. Second, they can be evaluated in parallel with a runtime that is logarithmic in the input length given enough processors; and they can also be evaluated sequentially. Third, their state and activations are bounded uniformly for all input lengths. Fourth, at almost all points, their loss gradient exists and it is bounded. Fifth, they do not exhibit a vanishing state gradient: the gradient of a state w.r.t. a past state can have constant value one regardless of the time distance between the two states. Finally, we find empirical evidence that the favourable theoretical properties of MinMax RNCs are matched by their practical capabilities: they are able to perfectly solve a number of synthetic tasks, showing superior performance compared to the considered state-of-the-art recurrent neural networks; also, we train a MinMax RNC of 127M parameters on next-token prediction, and the obtained model shows competitive performance for its size, providing evidence of the potential of MinMax RNCs on real-world tasks.