DeCAL Tokenwise Compression

TL;DR

DeCAL is a tokenwise compression method that uses a denoising pretrained encoder-decoder model to produce high-quality compressed representations, achieving comparable performance to uncompressed models at 2x compression and minor drops up to 8x across various NLP tasks.

Contribution

DeCAL introduces a novel tokenwise compression technique with minimal encoder modifications, enabling efficient dense representations without significant performance loss.

Findings

DeCAL at 2x compression matches uncompressed performance on several tasks.

Up to 8x compression, only minor metric drops observed.

Significant computational savings for dense representation tasks.

Abstract

This paper introduces DeCAL, a new method for tokenwise compression. DeCAL uses an encoder-decoder language model pretrained with denoising to learn to produce high-quality, general-purpose compressed representations from the encoder. DeCAL applies small modifications to the encoder, with the emphasis on maximizing compression quality, even at the expense of compute. We show that DeCAL at 2x compression can match uncompressed on several downstream tasks, with usually only a minor dropoff in metrics up to 8x compression, among question-answering, summarization, and multi-vector retrieval tasks. DeCAL offers significant savings where pre-computed dense representations can be utilized, and we believe the approach can be further developed to be more broadly applicable.