Representation Degeneration Problem in Training Natural Language Generation Models

TL;DR

This paper identifies a representation degeneration issue in training neural language models with weight tying, and proposes a regularization technique to improve embedding diversity and model performance.

Contribution

The paper introduces a novel regularization method that mitigates representation degeneration in neural language models trained with weight tying.

Findings

Regularization reduces embedding degeneration.

Improved language modeling and translation performance.

Embeddings maintain diversity with the proposed method.

Abstract

We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.