Small Character Models Match Large Word Models for Autocomplete Under Memory Constraints

TL;DR

This paper demonstrates that small character-based language models can match larger word-based models in autocomplete accuracy under memory constraints, especially in open-domain scenarios, by leveraging novel techniques and inductive biases.

Contribution

The study shows character models can rival word models in accuracy for autocomplete tasks under limited memory, introducing methods to enhance character models with compositional bias and transfer learning.

Findings

Character models perform similarly to larger word models in accuracy.

Character models are more memory-efficient for edge devices.

Novel techniques improve character model performance.

Abstract

Autocomplete is a task where the user inputs a piece of text, termed prompt, which is conditioned by the model to generate semantically coherent continuation. Existing works for this task have primarily focused on datasets (e.g., email, chat) with high frequency user prompt patterns (or focused prompts) where word-based language models have been quite effective. In this work, we study the more challenging open-domain setting consisting of low frequency user prompt patterns (or broad prompts, e.g., prompt about 93rd academy awards) and demonstrate the effectiveness of character-based language models. We study this problem under memory-constrained settings (e.g., edge devices and smartphones), where character-based representation is effective in reducing the overall model size (in terms of parameters). We use WikiText-103 benchmark to simulate broad prompts and demonstrate that character models rival word models in exact match accuracy for the autocomplete task, when controlled for the model size. For instance, we show that a 20M parameter character model performs similar to an 80M parameter word model in the vanilla setting. We further propose novel methods to improve character models by incorporating inductive bias in the form of compositional information and representation transfer from large word models. Datasets and code used in this work are available at https://github.com/UBC-NLP/char_autocomplete.