Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks

TL;DR

This paper introduces a framework to quantify and control solution degeneracy in task-trained RNNs, revealing how factors like task complexity and regularization influence internal variability across behavior, neural dynamics, and weights.

Contribution

It provides a systematic method to analyze and manipulate solution degeneracy in RNNs across multiple levels, validated on a large set of trained networks.

Findings

Higher task complexity reduces degeneracy in neural dynamics.

Larger networks and regularization decrease degeneracy across all levels.

Regularization and network size influence the variability of RNN solutions.

Abstract

Task-trained recurrent neural networks (RNNs) are widely used in neuroscience and machine learning to model dynamical computations. To gain mechanistic insight into how neural systems solve tasks, prior work often reverse-engineers individual trained networks. However, different RNNs trained on the same task and achieving similar performance can exhibit strikingly different internal solutions, a phenomenon known as solution degeneracy. Here, we develop a unified framework to systematically quantify and control solution degeneracy across three levels: behavior, neural dynamics, and weight space. We apply this framework to 3,400 RNNs trained on four neuroscience-relevant tasks: flip-flop memory, sine wave generation, delayed discrimination, and path integration, while systematically varying task complexity, learning regime, network size, and regularization. We find that higher task complexity and stronger feature learning reduce degeneracy in neural dynamics but increase it in weight space, with mixed effects on behavior. In contrast, larger networks and structural regularization reduce degeneracy at all three levels. These findings empirically validate the Contravariance Principle and provide practical guidance for researchers seeking to tune the variability of RNN solutions, either to uncover shared neural mechanisms or to model the individual variability observed in biological systems. This work provides a principled framework for quantifying and controlling solution degeneracy in task-trained RNNs, offering new tools for building more interpretable and biologically grounded models of neural computation.