Loop Neural Networks for Parameter Sharing

TL;DR

This paper introduces Loop Neural Networks that enhance language model performance by iterative refinement without increasing model size or requiring additional training data.

Contribution

It presents a novel looping mechanism for neural networks that allows for iterative prediction refinement, improving language modeling performance.

Findings

Loop models outperform standard GPT-2 in language tasks

Performance gains achieved without extra training data

Model size remains comparable to baseline GPT-2

Abstract

The success of large-scale language models like GPT can be attributed to their ability to efficiently predict the next token in a sequence. However, these models rely on constant computational effort regardless of the complexity of the token they are predicting, lacking the capacity for iterative refinement. In this paper, we introduce a novel Loop Neural Network, which achieves better performance by utilizing longer computational time without increasing the model size. Our approach revisits the input multiple times, refining the prediction by iteratively looping over a subset of the model with residual connections. We demonstrate the effectiveness of this method through experiments comparing versions of GPT-2 with our loop models, showing improved performance in language modeling tasks while maintaining similar parameter counts. Importantly, these improvements are achieved without the need for extra training data.