# Gated recurrent units viewed through the lens of continuous time dynamical systems

## TL;DR

This paper analyzes the dynamics of gated recurrent units (GRUs) using continuous time methods, revealing complex behaviors like oscillations and bifurcations, but not continuous attractors, thus deepening understanding of their capabilities.

## Contribution

It introduces a continuous time analysis of GRUs, uncovering diverse dynamical features and providing insights into their neural network behavior and biological plausibility.

## Key findings

- GRUs exhibit stable limit cycles and multi-stable dynamics
- They do not produce continuous attractors in low-dimensional settings
- The analysis offers new intuition on GRU internal mechanisms

## Abstract

Gated recurrent units (GRUs) are specialized memory elements for building recurrent neural networks. Despite their incredible success on various tasks, including extracting dynamics underlying neural data, little is understood about the specific dynamics representable in a GRU network. As a result, it is both difficult to know a priori how successful a GRU network will perform on a given task, and also their capacity to mimic the underlying behavior of their biological counterparts. Using a continuous time analysis, we gain intuition on the inner workings of GRU networks. We restrict our presentation to low dimensions, allowing for a comprehensive visualization. We found a surprisingly rich repertoire of dynamical features that includes stable limit cycles (nonlinear oscillations), multi-stable dynamics with various topologies, and homoclinic bifurcations. At the same time we were unable to train GRU networks to produce continuous attractors, which are hypothesized to exist in biological neural networks. We contextualize the usefulness of different kinds of observed dynamics and support our claims experimentally.