The Power of External Memory in Increasing Predictive Model Capacity

TL;DR

This paper explores how external memory tables can enhance deep network capacity by different lookup and consumption methods, introduces a new alternating updates technique, and demonstrates its effectiveness in language modeling.

Contribution

It provides a comprehensive experimental evaluation of existing external memory methods and introduces a novel alternating updates approach for improved language modeling.

Findings

External memory tables increase model capacity without extra inference cost.

Alternating updates enable larger token dimensions efficiently.

The new method improves language modeling performance.

Abstract

One way of introducing sparsity into deep networks is by attaching an external table of parameters that is sparsely looked up at different layers of the network. By storing the bulk of the parameters in the external table, one can increase the capacity of the model without necessarily increasing the inference time. Two crucial questions in this setting are then: what is the lookup function for accessing the table and how are the contents of the table consumed? Prominent methods for accessing the table include 1) using words/wordpieces token-ids as table indices, 2) LSH hashing the token vector in each layer into a table of buckets, and 3) learnable softmax style routing to a table entry. The ways to consume the contents include adding/concatenating to input representation, and using the contents as expert networks that specialize to different inputs. In this work, we conduct rigorous experimental evaluations of existing ideas and their combinations. We also introduce a new method, alternating updates, that enables access to an increased token dimension without increasing the computation time, and demonstrate its effectiveness in language modeling.