Contraction Properties of the Global Workspace Primitive

TL;DR

This paper advances the theoretical understanding and empirical performance of multi-area recurrent neural networks with a focus on global workspace structures, demonstrating improved stability and robustness.

Contribution

It provides relaxed stability conditions for global workspace RNNs and empirically shows their effectiveness with fewer parameters and increased resilience.

Findings

Global workspace RNNs achieve strong test performance.

Empirical resilience to subnetwork removal.

Improved performance on benchmark sequence tasks.

Abstract

To push forward the important emerging research field surrounding multi-area recurrent neural networks (RNNs), we expand theoretically and empirically on the provably stable RNNs of RNNs introduced by Kozachkov et al. in "RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks". We prove relaxed stability conditions for salient special cases of this architecture, most notably for a global workspace modular structure. We then demonstrate empirical success for Global Workspace Sparse Combo Nets with a small number of trainable parameters, not only through strong overall test performance but also greater resilience to removal of individual subnetworks. These empirical results for the global workspace inter-area topology are contingent on stability preservation, highlighting the relevance of our theoretical work for enabling modular RNN success. Further, by exploring sparsity in the connectivity structure between different subnetwork modules more broadly, we improve the state of the art performance for stable RNNs on benchmark sequence processing tasks, thus underscoring the general utility of specialized graph structures for multi-area RNNs.