The Lower The Simpler: Simplifying Hierarchical Recurrent Models

TL;DR

This paper introduces a strategy called 'the lower the simpler' to simplify hierarchical recurrent models by replacing complex layers with simpler variants, leading to faster training and improved performance.

Contribution

The paper proposes a novel simplification strategy for hierarchical recurrent models, including new variants like SGU and FOFE, to enhance training efficiency without performance loss.

Findings

Simplified models have fewer parameters.

Training time is significantly reduced.

Performance is slightly improved.

Abstract

To improve the training efficiency of hierarchical recurrent models without compromising their performance, we propose a strategy named as `the lower the simpler', which is to simplify the baseline models by making the lower layers simpler than the upper layers. We carry out this strategy to simplify two typical hierarchical recurrent models, namely Hierarchical Recurrent Encoder-Decoder (HRED) and R-NET, whose basic building block is GRU. Specifically, we propose Scalar Gated Unit (SGU), which is a simplified variant of GRU, and use it to replace the GRUs at the middle layers of HRED and R-NET. Besides, we also use Fixed-size Ordinally-Forgetting Encoding (FOFE), which is an efficient encoding method without any trainable parameter, to replace the GRUs at the bottom layers of HRED and R-NET. The experimental results show that the simplified HRED and the simplified R-NET contain significantly less trainable parameters, consume significantly less training time, and achieve slightly better performance than their baseline models.