Hierarchical Conflict Propagation: Sequence Learning in a Recurrent Deep Neural Network

TL;DR

This paper introduces a novel training method for recurrent neural networks that uses parallel cloned networks to hierarchically propagate conflicts, improving learning of long-term dependencies in sequence tasks.

Contribution

A new training approach with parallel clones that enables hierarchical conflict propagation, enhancing RNNs' ability to learn long-term dependencies.

Findings

Effective in memorizing long sequences like Moby Dick paragraph

Hierarchical conflict propagation improves long-term dependency learning

Parallel clones method outperforms traditional training in sequence tasks

Abstract

Recurrent neural networks (RNN) are capable of learning to encode and exploit activation history over an arbitrary timescale. However, in practice, state of the art gradient descent based training methods are known to suffer from difficulties in learning long term dependencies. Here, we describe a novel training method that involves concurrent parallel cloned networks, each sharing the same weights, each trained at different stimulus phase and each maintaining independent activation histories. Training proceeds by recursively performing batch-updates over the parallel clones as activation history is progressively increased. This allows conflicts to propagate hierarchically from short-term contexts towards longer-term contexts until they are resolved. We illustrate the parallel clones method and hierarchical conflict propagation with a character-level deep RNN tasked with memorizing a paragraph of Moby Dick (by Herman Melville).