pNLP-Mixer: an Efficient all-MLP Architecture for Language

TL;DR

The paper introduces pNLP-Mixer, a compact all-MLP architecture for on-device NLP tasks that achieves high performance with significantly fewer parameters than transformer models, making it suitable for constrained devices.

Contribution

It proposes a novel embedding-free MLP-Mixer model with a unique projection layer, enabling high weight-efficiency for on-device NLP applications.

Findings

Achieves 99.4% and 97.8% of mBERT performance on two datasets.

Uses 170x fewer parameters than mBERT.

Outperforms state-of-the-art tiny models by up to 7.8%.

Abstract

Large pre-trained language models based on transformer architecture have drastically changed the natural language processing (NLP) landscape. However, deploying those models for on-device applications in constrained devices such as smart watches is completely impractical due to their size and inference cost. As an alternative to transformer-based architectures, recent work on efficient NLP has shown that weight-efficient models can attain competitive performance for simple tasks, such as slot filling and intent classification, with model sizes in the order of the megabyte. This work introduces the pNLP-Mixer architecture, an embedding-free MLP-Mixer model for on-device NLP that achieves high weight-efficiency thanks to a novel projection layer. We evaluate a pNLP-Mixer model of only one megabyte in size on two multi-lingual semantic parsing datasets, MTOP and multiATIS. Our quantized model achieves 99.4% and 97.8% the performance of mBERT on MTOP and multi-ATIS, while using 170x fewer parameters. Our model consistently beats the state-of-the-art of tiny models (pQRNN), which is twice as large, by a margin up to 7.8% on MTOP.