Superbloom: Bloom filter meets Transformer

TL;DR

This paper introduces a novel approach combining Bloom filter-inspired hashing with Transformer models to efficiently handle large vocabularies in machine learning tasks, achieving high accuracy with smaller models.

Contribution

It presents a new method that applies multi-layer Transformers to Bloom filter digests for improved performance on large vocabulary tasks.

Findings

Models outperform similar-sized non-hashed models

Models rival larger models trained with sampled softmax

Multi-layer Transformer reduces ambiguity in hashed inputs

Abstract

We extend the idea of word pieces in natural language models to machine learning tasks on opaque ids. This is achieved by applying hash functions to map each id to multiple hash tokens in a much smaller space, similarly to a Bloom filter. We show that by applying a multi-layer Transformer to these Bloom filter digests, we are able to obtain models with high accuracy. They outperform models of a similar size without hashing and, to a large degree, models of a much larger size trained using sampled softmax with the same computational budget. Our key observation is that it is important to use a multi-layer Transformer for Bloom filter digests to remove ambiguity in the hashed input. We believe this provides an alternative method to solving problems with large vocabulary size.