Tiny Neural Models for Seq2Seq

TL;DR

This paper introduces a tiny, efficient seq2seq model called pQRNN-MAtt for semantic parsing, demonstrating superior performance on a multilingual dataset while being significantly smaller and suitable for on-device applications.

Contribution

It extends projection-based methods to seq2seq architectures, creating models under 3.5MB that outperform larger LSTM models and can distill large pre-trained models.

Findings

Model size under 3.5MB surpasses larger LSTM models.

Achieves better performance on MTOP dataset.

Effective as a student for distilling T5/BERT.

Abstract

Semantic parsing models with applications in task oriented dialog systems require efficient sequence to sequence (seq2seq) architectures to be run on-device. To this end, we propose a projection based encoder-decoder model referred to as pQRNN-MAtt. Studies based on projection methods were restricted to encoder-only models, and we believe this is the first study extending it to seq2seq architectures. The resulting quantized models are less than 3.5MB in size and are well suited for on-device latency critical applications. We show that on MTOP, a challenging multilingual semantic parsing dataset, the average model performance surpasses LSTM based seq2seq model that uses pre-trained embeddings despite being 85x smaller. Furthermore, the model can be an effective student for distilling large pre-trained models such as T5/BERT.