Input-to-Output Gate to Improve RNN Language Models

TL;DR

This paper introduces the Input-to-Output Gate (IOG), a simple reinforcement method that enhances RNN language models' performance by refining their output layers, demonstrated on Penn Treebank and WikiText-2 datasets.

Contribution

The paper presents a novel, simple gating mechanism called IOG that can be integrated with existing RNN language models to improve their performance.

Findings

IOG consistently improves RNN language model performance.

Effective across different RNN architectures.

Demonstrated on Penn Treebank and WikiText-2 datasets.

Abstract

This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.