Efficiency through Auto-Sizing: Notre Dame NLP's Submission to the WNGT 2019 Efficiency Task

TL;DR

This paper explores auto-sizing techniques applied to Transformer models to significantly reduce parameters with minimal performance loss, demonstrating over 25% parameter reduction and only a 1.1 BLEU decrease.

Contribution

The paper introduces an auto-sizing approach tailored for Transformers, achieving substantial parameter reduction while maintaining competitive performance.

Findings

Over 25% parameter reduction achieved

Only 1.1 BLEU decrease observed

Auto-sizing effectively reduces model size

Abstract

This paper describes the Notre Dame Natural Language Processing Group's (NDNLP) submission to the WNGT 2019 shared task (Hayashi et al., 2019). We investigated the impact of auto-sizing (Murray and Chiang, 2015; Murray et al., 2019) to the Transformer network (Vaswani et al., 2017) with the goal of substantially reducing the number of parameters in the model. Our method was able to eliminate more than 25% of the model's parameters while suffering a decrease of only 1.1 BLEU.