Natural Language Processing with Small Feed-Forward Networks

TL;DR

This paper demonstrates that small, shallow feed-forward neural networks can perform competitively on language processing tasks, offering a resource-efficient alternative to deep models especially for mobile and low-resource environments.

Contribution

It introduces techniques for designing small neural networks that balance performance and resource constraints in language processing applications.

Findings

Small networks achieve near state-of-the-art results

Memory and computational costs are significantly reduced

Tradeoffs in model size and accuracy are analyzed

Abstract

We show that small and shallow feed-forward neural networks can achieve near state-of-the-art results on a range of unstructured and structured language processing tasks while being considerably cheaper in memory and computational requirements than deep recurrent models. Motivated by resource-constrained environments like mobile phones, we showcase simple techniques for obtaining such small neural network models, and investigate different tradeoffs when deciding how to allocate a small memory budget.