WaLDORf: Wasteless Language-model Distillation On Reading-comprehension

TL;DR

WaLDORf is a novel hybrid convolutional-transformer model that achieves state-of-the-art inference speed and improved accuracy for reading comprehension tasks, addressing the resource constraints of large language models.

Contribution

Introduces WaLDORf, a task-specific hybrid model that combines convolutional and transformer techniques for efficient and accurate reading comprehension.

Findings

WaLDORf outperforms previous distilled models in accuracy.

WaLDORf achieves faster inference speeds.

WaLDORf maintains high performance on reading comprehension tasks.

Abstract

Transformer based Very Large Language Models (VLLMs) like BERT, XLNet and RoBERTa, have recently shown tremendous performance on a large variety of Natural Language Understanding (NLU) tasks. However, due to their size, these VLLMs are extremely resource intensive and cumbersome to deploy at production time. Several recent publications have looked into various ways to distil knowledge from a transformer based VLLM (most commonly BERT-Base) into a smaller model which can run much faster at inference time. Here, we propose a novel set of techniques which together produce a task-specific hybrid convolutional and transformer model, WaLDORf, that achieves state-of-the-art inference speed while still being more accurate than previous distilled models.