Linear Memory Networks

TL;DR

The paper introduces the Linear Memory Network, a novel recurrent architecture that separates the functional transformation from the memory mechanism, enabling efficient training and competitive performance on music sequence tasks.

Contribution

It presents a new recurrent model with separate components for transformation and memory, using linear autoencoders and closed-form training solutions.

Findings

Achieves competitive results on polyphonic music datasets.

Efficient training via closed-form solutions.

Pretraining schema improves performance.

Abstract

Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation between the functional input-output transformation and the memory mechanism, showing how they can be implemented through different neural components. By building on such conceptualization, we introduce the Linear Memory Network, a recurrent model comprising a feedforward neural network, realizing the non-linear functional transformation, and a linear autoencoder for sequences, implementing the memory component. The resulting architecture can be efficiently trained by building on closed-form solutions to linear optimization problems. Further, by exploiting equivalence results between feedforward and recurrent neural networks we devise a pretraining schema for the proposed architecture. Experiments on polyphonic music datasets show competitive results against gated recurrent networks and other state of the art models.