HashFormers: Towards Vocabulary-independent Pre-trained Transformers

TL;DR

HashFormers introduce a vocabulary-independent pre-trained transformer architecture that uses hashing to significantly reduce memory usage while maintaining competitive performance on text classification tasks.

Contribution

This work presents HashFormers, a novel pre-trained transformer model that employs hashing functions to eliminate the need for large embedding matrices, enabling unlimited vocabulary support.

Findings

HashFormers are more memory efficient than standard models.

They achieve comparable performance on text classification tasks.

The most efficient variant uses only 99.1K parameters with minimal performance loss.

Abstract

Transformer-based pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. This one-to-one mapping results into embedding matrices that occupy a lot of memory (i.e. millions of parameters) and grow linearly with the size of the vocabulary. Previous work on on-device transformers dynamically generate token embeddings on-the-fly without embedding matrices using locality-sensitive hashing over morphological information. These embeddings are subsequently fed into transformer layers for text classification. However, these methods are not pre-trained. Inspired by this line of work, we propose HashFormers, a new family of vocabulary-independent pre-trained transformers that support an unlimited vocabulary (i.e. all possible tokens in a corpus) given a substantially smaller fixed-sized embedding matrix. We achieve this by first introducing computationally cheap hashing functions that bucket together individual tokens to embeddings. We also propose three variants that do not require an embedding matrix at all, further reducing the memory requirements. We empirically demonstrate that HashFormers are more memory efficient compared to standard pre-trained transformers while achieving comparable predictive performance when fine-tuned on multiple text classification tasks. For example, our most efficient HashFormer variant has a negligible performance degradation (0.4\% on GLUE) using only 99.1K parameters for representing the embeddings compared to 12.3-38M parameters of state-of-the-art models.