Flow Equivariant Recurrent Neural Networks

TL;DR

This paper introduces flow equivariant recurrent neural networks that respect continuous symmetries over time, improving generalization and training efficiency for sequence modeling tasks involving dynamic transformations.

Contribution

It extends equivariant network theory to RNNs for continuous flow transformations, a novel approach for sequence models respecting time-parameterized symmetries.

Findings

Flow equivariant RNNs outperform standard RNNs in training speed.

They generalize better to longer sequences and different velocities.

The approach improves sequence prediction and classification tasks.

Abstract

Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of 'flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.