General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings

TL;DR

This paper introduces a novel method for distilling large pretrained language models into a more efficient matrix embedding architecture, achieving competitive performance with significantly fewer parameters and faster inference.

Contribution

It presents a new cross-architecture distillation approach into a matrix-based model, extending CMOW with bidirectional and task-agnostic features, improving efficiency and performance.

Findings

Competitive results to DistilBERT on question similarity and textual entailment.

Uses half the parameters and is three times faster in inference.

Doubling of scores on linguistic acceptability tasks compared to previous methods.

Abstract

Large pretrained language models (PreLMs) are revolutionizing natural language processing across all benchmarks. However, their sheer size is prohibitive for small laboratories or for deployment on mobile devices. Approaches like pruning and distillation reduce the model size but typically retain the same model architecture. In contrast, we explore distilling PreLMs into a different, more efficient architecture, Continual Multiplication of Words (CMOW), which embeds each word as a matrix and uses matrix multiplication to encode sequences. We extend the CMOW architecture and its CMOW/CBOW-Hybrid variant with a bidirectional component for more expressive power, per-token representations for a general (task-agnostic) distillation during pretraining, and a two-sequence encoding scheme that facilitates downstream tasks on sentence pairs, such as sentence similarity and natural language inference. Our matrix-based bidirectional CMOW/CBOW-Hybrid model is competitive to DistilBERT on question similarity and recognizing textual entailment, but uses only half of the number of parameters and is three times faster in terms of inference speed. We match or exceed the scores of ELMo for all tasks of the GLUE benchmark except for the sentiment analysis task SST-2 and the linguistic acceptability task CoLA. However, compared to previous cross-architecture distillation approaches, we demonstrate a doubling of the scores on detecting linguistic acceptability. This shows that matrix-based embeddings can be used to distill large PreLM into competitive models and motivates further research in this direction.