On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression

TL;DR

This paper investigates applying low-rank matrix factorization to compress LSTM networks, reducing computational costs while maintaining performance across various NLP tasks.

Contribution

It introduces a novel application of low-rank matrix factorization to LSTM recurrences, highlighting the importance of additive over multiplicative recurrence compression.

Findings

Additive recurrence is more crucial than multiplicative recurrence.

Matrix norm correlations explain compression effectiveness.

Effective LSTM compression achieved in language models and ELMo.

Abstract

Despite their ubiquity in NLP tasks, Long Short-Term Memory (LSTM) networks suffer from computational inefficiencies caused by inherent unparallelizable recurrences, which further aggravates as LSTMs require more parameters for larger memory capacity. In this paper, we propose to apply low-rank matrix factorization (MF) algorithms to different recurrences in LSTMs, and explore the effectiveness on different NLP tasks and model components. We discover that additive recurrence is more important than multiplicative recurrence, and explain this by identifying meaningful correlations between matrix norms and compression performance. We compare our approach across two settings: 1) compressing core LSTM recurrences in language models, 2) compressing biLSTM layers of ELMo evaluated in three downstream NLP tasks.